How does the Orthodox Church combine the use of the two main books of hymnody for the order of weekday services of Vespers?
We are dealing, in this question, ONLY WITH MONDAY THROUGH FRIDAY. Saturday and Sunday have their own rules.
We are also dealing with the periods wherein there is NONE of the following:
1. Period of FOREFEAST (leading up to a great feast)
2. Period of AFTERFEAST (culminating in an apodosis of a great feast)
3. The entire period of the TRIODION and the PENTEKOSTARION; namely, from Monday of Cheesefare Week through the Friday after Pentecost.
First, go to the Menaion for the given date:
1. Are there THREE DOXASTIKA at Vespers? (at the lamp-lighting, at the litia, and at the aposticha). If so, you will NOT use the Octoechos.
2. Is the service in the Menaion a single or double commemoration?
a. If a single commemoration, you will chant THREE stichera at the lamp-lighting FROM THE OCTOECHOS OF THE DAY AND IN THE MODE OF THE WEEK, followed by the three stichera for the saint in the Menaion.
NOTE CAREFULLY: WHAT IS EXACTLY CHANTED FROM THE OCTOECHOS? The three taken from the Octoechos are as follows–the first two from the first set of three, and the first one of the second set of three.
If there be a doxastikon in the Menaion, then chant that; otherwise after Glory… Both now…, the theotokion (on Sunday, Monday, Wednesday evenings), or stavro-theotokion (on Tuesday and Thursday evenings) as given in the Menaion.
b. If a double commemoration, you will chant ALL SIX from the Menaion, omitting the Octoechos at the lamp-lighting. (there are exceptions to this, however, for generic purposes, this rule suffices).
3. Frequently, the Menaion provides only a doxastikon and theotokion and stavro-theotokion for the Aposticha of Vespers. When this happens, the three stichera from the Octoechos for the day of the week in the mode of the week are chanted, along with their usual verses. Then, Glory… doxastikon, Both now… and (stavro-)theotokion are taken from the Menaion, omitting that which is given in the Octoechos.
4. WHAT APOLYTIKION AND THEOTOKION IS CHANTED?
a. If there are NO doxastika for vespers given in the Menaion, then NO apolytikion for the saint is chanted. Instead, the apolytikion for the day, along with its theotokion, is chanted. These are found in the Great Horologion (HTM, Boston), pages 643-645.
b. If the given saint of the day is festally commemorated (having at least 2 doxastika and the Great Doxology at Orthros the next day), then the apolytikion of the saint (or both of the saints, as may be called for) is chanted, followed by the so-called “resurrection” theotokion in the matching mode. These theotokia are provided in the Great Horologion next to the resurrectional apolytikia, one for each mode, on pages 637-642.
c. If the given saint of the day is a simple commemoration, but has at least one doxastikon (thus, NO Great Doxology is called for at Orthros the next morning), then the apolytikion of the saint (or saints, if double) is chanted, followed by the simple theotokion of the SAME mode as the apolytikion, for the day of the week. These theotokia are found in the Great Horologion on pages 646-660.
d. In a double commemoration, if there are stichera for both saints at the lamp-lighting, but only doxastika for ONE of those saints, then we chant ONLY the apolytikion of the saint bearing a doxastikon.
WHAT KIND OF DISMISSAL PRAYER IS SAID BY THE PRIEST AT THE END?
In all cases, for Vespers (unlike the Hours), the priest will say the Great Dismissal. This is commonly misunderstood, so I back up my assertion by citing an authoritative reference: Diptycha of the Church of Greece 2009, p. mb (Greek numerals; = xlii), published by Apostolike Diakonia. This is the authoritative typikon for my diocese.