A rectoscopy is one of many components under endoscopy, which basically means looking inside the body for medical reasons. The instrument used is called an endoscope. This device goes inside the organ to properly observe from the inside.
A rectoscopy specifically is the endoscopy of the rectum, which is an organ used in defecation. Essentially the endoscope is inserted into a place where the "sun don't shine," so to speak.