Short answer
Mary was accused of adultery long before Islam. The charge appears in the pagan writer Celsus and in the Babylonian Talmud. The Qur'an calls it "a grave false charge" (4:156). About 175-180 CE, Celsus has a Jewish speaker say that Jesus' mother "bore a child to a certain soldier named Panthera". The Tosefta, a rabbinic collection of about the third century, speaks of "Yeshua ben Pantera" but not of his mother. The Babylonian Talmud tells of a man called Ben Stada or Ben Pandira, whose mother was named Miriam. A Babylonian saying explains her nickname: she "strayed from her husband". One manuscript of the Talmud names him "Jesus the Nazarene". The Qur'an answers the charge. God (﷾) chose Mary and purified her (3:42). Her son was created by His command, as Adam was (3:59).
Full explanation
The earliest record: Celsus' Jewish speaker (about 175-180 CE)
Celsus was a pagan philosopher who wrote a book against Christianity, The True Word, about 175-180 CE. It is lost. But Origen, a Christian scholar, quoted it at length in his reply, written about 248. In one part Celsus brings in a Jewish speaker who argues with Jesus. His first charge is that Jesus invented his birth from a virgin. Later he gives his own account of the birth and names the father Panthera. So the charge was being made against the Christians' Jesus, by name, within about 150 years of his life.
“For he represents him disputing with Jesus, and confuting Him, as he thinks, on many points; and in the first place, he accuses Him of having "invented his birth from a virgin,"”
“But let us now return to where the Jew is introduced, speaking of the mother of Jesus, and saying that "when she was pregnant she was turned out of doors by the carpenter to whom she had been betrothed, as having been guilty of adultery, and that she bore a child to a certain soldier named Panthera;"”
A rabbinic text names "Yeshua ben Pantera" (Tosefta Hullin 2:22-24)
The Tosefta is an early collection of rabbinic teaching, compiled in about the third century. In its section Hullin it gives a rule of Jewish law: do not deal with sectarians, called minim, and do not accept healing from them. Two stories follow. In the first, a man named Jacob came to heal Rabbi Eleazar ben Dama of a snakebite "in the name of Yeshua ben Pantera". Rabbi Ishmael, a sage of the early 2nd century, forbade it, and Ben Dama died. In the second, Rabbi Eliezer, a sage of about 100 CE, recalls a heretical teaching. A Jacob gave it to him "in the name of Yeshua ben Pantera". Healing and teaching in Jesus' name is what his followers did. So R. Travers Herford, an English scholar who translated these passages in 1903, judged that these texts plainly mean Jesus. The Tosefta says nothing about his mother.
Halakha with two stories (ma'aseh). The rule just before (2:21) forbids dealing with minim, sectarians, and being healed by them. 2:22-23: Rabbi Eleazar ben Dama was bitten by a snake. Jacob of Kefar Sama came to heal him "in the name of Yeshua ben Pantera" (mi-shum Yeshua ben Pantera). Rabbi Ishmael would not allow it, and Ben Dama died before he could bring his proof that it was permitted. Rabbi Ishmael praised him for not breaking "the fence of the Sages" (Ecclesiastes 10:8). 2:24: Rabbi Eliezer, arrested on a charge of minut (heresy) and released, tells Rabbi Akiva that in the street of Sepphoris Jacob of Kefar Sikhnin told him a word of minut "in the name of Yeshua ben Pantera", and it pleased him. Nothing is said here about Jesus' mother; the Tosefta gives only the name. The spelling of the second name varies between editions: Pantera in the vocalised text on Wikisource; Herford, translating Zuckermandel's edition, gives Pandira at 2:22 and Pantiri at 2:24.
“The case of R. El'azar ben Damah, whom a serpent bit. There came in Jacob, a man of Chephar Sama, to cure him in the name of Jeshua' ben Pandira, but R. Ishmael did not allow it. He said, 'Thou art not permitted, Ben Damah.' He said, 'I will bring thee a proof that he may heal me.' But he had not finished bringing a proof when he died.”
“Once I was walking along the street of Sepphoris, and I met Jacob of Chephar Sichnin, and he said to me a word of Minuth in the name of Jeshu ben Pantiri, and it pleased me.”
Herford translates Shabbat 104b and explains its steps. He holds that Ben Stada and Ben Pandira, as the Talmud uses them, are one person, and that the person is Jesus, because the Tosefta speaks of healing "in the name of Jeshu ben Pandira" (pp. 37-38). He calls the Pumbedita explanation of Stada "a mere guess" (p. 38). He notes that Pappos ben Jehudah lived a century after Jesus, and that megaddela is based on the name Magdala (p. 40). He judges that "only a very dim and confused notion existed as to the parentage of Jesus" when the passage was recorded (p. 40). He thinks the two names were old taunts against Jesus whose meaning is now lost (p. 40). He later allows that Ben Stada may first have meant another man, "that Egyptian" of Acts 21:38, and was only later confused with Jesus (p. 345 n. 1). He judges that these passages add nothing new to the history of Jesus (p. 359).
The Babylonian Talmud's story of Ben Stada
The story about the mother comes later, in the Babylonian Talmud, edited about 500-600 CE. It is told of a man called Ben Stada, or Ben Pandira. It appears twice, in the volumes (tractates) Shabbat, on page 104b, and Sanhedrin, on page 67a. In Shabbat 104b Rabbi Eliezer asks whether Ben Stada did not bring sorceries from Egypt in cuts on his flesh. The Sages, the early rabbis, answer: "He was a fool". In Sanhedrin 67a the law is about a man who entices others to worship idols. An early teaching names Ben Stada as an example of how such a man was caught.
Halakha on the Sabbath. The Mishnah here says that one who scratches marks on his flesh on the Sabbath is liable in Rabbi Eliezer's view and exempt in the Sages'. A baraita adds that Rabbi Eliezer said to the Sages: did not Ben Stada bring sorceries out of Egypt "in an incision on his flesh" (bi-seritah she-al besaro)? They answered: "He was a fool" (shoteh hayah), and proof is not brought from fools. The name Yeshu does not appear. The Gemara then debates who Ben Stada was and ties him to Ben Pandira and a mother named Miriam; that discussion belongs to the question of Mary.
Aggadah set inside a legal discussion. The Mishnah (Sanhedrin 7:10) deals with the enticer (mesit) who urges others to worship idols. A baraita says that for this offence alone the court hides witnesses to catch him, and adds: "And so they did to Ben Stada in Lud, and they hanged him on the eve of Passover." The Gemara then asks, as in Shabbat 104b: Ben Stada? He is Ben Pandira! Rav Hisda answers that Stada was the husband and Pandira another man (bo'el Pandira). The Gemara objects that the husband was Pappos ben Yehudah; perhaps the mother was Stada. It objects again that the mother was Miriam megaddela neshaya (here without se'ar, "hair", which in Shabbat 104b is found only in the Vilna print; Schäfer n. 11). It answers with a saying of Pumbedita: satat da mi-ba'alah, "this one strayed from her husband". The name Yeshu does not appear. The printed Vilna text has these lines only in Hesronot ha-Shas, the supplement of censored passages.
The mother's name and the Pumbedita saying
Both passages then ask why he is called Ben Stada and also Ben Pandira. Rav Hisda was a Babylonian teacher who died about 309. He suggests that Stada was the name of the mother's husband and Pandira the name of another man. The Talmud objects that the husband was Pappos ben Yehudah. It adds that the mother was Miriam megaddela neshaya, "who braided women's hair". It ends with a saying from the Babylonian town of Pumbedita: "this one strayed from her husband".
Halakha on the Sabbath. The Mishnah here says that one who scratches marks on his flesh on the Sabbath is liable in Rabbi Eliezer's view and exempt in the Sages'. A baraita adds that Rabbi Eliezer said to the Sages: did not Ben Stada bring sorceries out of Egypt "in an incision on his flesh" (bi-seritah she-al besaro)? They answered: "He was a fool" (shoteh hayah), and proof is not brought from fools. The name Yeshu does not appear. The Gemara then debates who Ben Stada was and ties him to Ben Pandira and a mother named Miriam; that discussion belongs to the question of Mary.
Aggadah set inside a legal discussion. The Mishnah (Sanhedrin 7:10) deals with the enticer (mesit) who urges others to worship idols. A baraita says that for this offence alone the court hides witnesses to catch him, and adds: "And so they did to Ben Stada in Lud, and they hanged him on the eve of Passover." The Gemara then asks, as in Shabbat 104b: Ben Stada? He is Ben Pandira! Rav Hisda answers that Stada was the husband and Pandira another man (bo'el Pandira). The Gemara objects that the husband was Pappos ben Yehudah; perhaps the mother was Stada. It objects again that the mother was Miriam megaddela neshaya (here without se'ar, "hair", which in Shabbat 104b is found only in the Vilna print; Schäfer n. 11). It answers with a saying of Pumbedita: satat da mi-ba'alah, "this one strayed from her husband". The name Yeshu does not appear. The printed Vilna text has these lines only in Hesronot ha-Shas, the supplement of censored passages.
One manuscript names "Jesus the Nazarene"
One manuscript, MS Vatican ebr. 108 (13th-14th century), adds in Shabbat 104b that he is "Jesus the Nazarene". Peter Schäfer, a modern scholar of these passages, reports this. The printed Vilna edition, the standard text today, carries these lines only in Hesronot ha-Shas. That is the supplement that preserved what Church censorship removed.
Halakha on the Sabbath. The Mishnah here says that one who scratches marks on his flesh on the Sabbath is liable in Rabbi Eliezer's view and exempt in the Sages'. A baraita adds that Rabbi Eliezer said to the Sages: did not Ben Stada bring sorceries out of Egypt "in an incision on his flesh" (bi-seritah she-al besaro)? They answered: "He was a fool" (shoteh hayah), and proof is not brought from fools. The name Yeshu does not appear. The Gemara then debates who Ben Stada was and ties him to Ben Pandira and a mother named Miriam; that discussion belongs to the question of Mary.
Schäfer argues that the Babylonian passage (Shabbat 104b, Sanhedrin 67a) does refer to the Jesus of the New Testament, though its text does not name him. He notes, as scholars long have, "a remarkable parallel" in the Jew of Celsus (p. 18), and concludes that both most likely draw on common, originally Jewish sources. He reads it as a counter-narrative that inverts the birth stories of Matthew and Luke. He answers Johann Maier, who suggested another man called Pantera: possible, but not very likely, because the name is not common in Hebrew or Aramaic. He reports that only one manuscript, Vatican ebr. 108 (13th-14th century), names the child "Jesus the Nazarene". In a note he suggests the Talmud may conflate Mary the mother of Jesus with Mary Magdalene, citing B. Visotzky's reading of megaddela.
Who is Ben Stada?
Rashi, an 11th-century Jewish commentator (d. 1105), read the passage as a charge against the mother. His notes on it were later removed by the censor. Rabbenu Tam, a leading 12th-century rabbi (d. 1171), denied that Ben Stada was Yeshu ha-Notzri, Jesus the Nazarene. He argued from dates: Pappos ben Yehudah lived in Rabbi Akiva's time, in the 2nd century. Modern scholars differ too. Herford (1903) held that Ben Stada and Ben Pandira, as the Talmud uses them, mean Jesus. Peter Schäfer (2007) agrees that the passage means the Jesus of the New Testament Gospels. He notes "a remarkable parallel" in the Jewish speaker in Celsus. He thinks both drew on common, most likely Jewish sources. He reads the story as a deliberate reversal of the birth stories of Matthew and Luke. Herford and Schäfer both note that the Talmud's Miriam megaddela may merge Mary the mother of Jesus with Mary Magdalene. Johann Maier, a scholar of Judaism writing in 1978, argued that the name was attached to Jesus only late. Joseph Klausner (1922), a Jewish historian, held that Ben Stada was another man. He still took Ben Pandera to be an old name for Jesus.
On Rav Hisda's words about the husband and Pandira, Rashi explains that the son was named after his mother's husband, although he was not the husband's child. So Rashi read the passage as the charge of adultery against the mother.
Rashi explains "Ben Stada" as named after his mother's husband, Stada, and the Pumbedita phrase satat da as a nickname that refers to the charge against her.
On "Ben Stada", uncensored: Rabbenu Tam said that this is not Yeshu ha-Notzri. The Ben Stada spoken of here lived in the days of Pappos ben Yehudah, who lived in the days of Rabbi Akiva (Berakhot 61b), while Yeshu lived in the days of Yehoshua ben Perachiah (Sotah 47a), long before Rabbi Akiva. The Tosafot add that the "Miriam megaddela neshaya" of Hagigah 4b, in the time of Rav Bibi, was another woman, or else that the angel of death was telling Rav Bibi of something that had happened long before (i nami mal'akh ha-mavet hayah mesaper).
Herford translates Shabbat 104b and explains its steps. He holds that Ben Stada and Ben Pandira, as the Talmud uses them, are one person, and that the person is Jesus, because the Tosefta speaks of healing "in the name of Jeshu ben Pandira" (pp. 37-38). He calls the Pumbedita explanation of Stada "a mere guess" (p. 38). He notes that Pappos ben Jehudah lived a century after Jesus, and that megaddela is based on the name Magdala (p. 40). He judges that "only a very dim and confused notion existed as to the parentage of Jesus" when the passage was recorded (p. 40). He thinks the two names were old taunts against Jesus whose meaning is now lost (p. 40). He later allows that Ben Stada may first have meant another man, "that Egyptian" of Acts 21:38, and was only later confused with Jesus (p. 345 n. 1). He judges that these passages add nothing new to the history of Jesus (p. 359).
Schäfer argues that the Babylonian passage (Shabbat 104b, Sanhedrin 67a) does refer to the Jesus of the New Testament, though its text does not name him. He notes, as scholars long have, "a remarkable parallel" in the Jew of Celsus (p. 18), and concludes that both most likely draw on common, originally Jewish sources. He reads it as a counter-narrative that inverts the birth stories of Matthew and Luke. He answers Johann Maier, who suggested another man called Pantera: possible, but not very likely, because the name is not common in Hebrew or Aramaic. He reports that only one manuscript, Vatican ebr. 108 (13th-14th century), names the child "Jesus the Nazarene". In a note he suggests the Talmud may conflate Mary the mother of Jesus with Mary Magdalene, citing B. Visotzky's reading of megaddela.
Argues that no early rabbinic passage clearly refers to Jesus, and that the passages that name him, Sanhedrin 43a and 107b among them, received the name late, largely after the Talmud was completed.
A Jewish historian. He sets aside the Talmud's later (amoraic) statements about Jesus and the Ben Stada passages, which he holds are not about Jesus, and keeps a few early (tannaitic) ones. Among them he gives the Sanhedrin 43a baraita more historical weight than the rest. He holds that its date, the eve of Passover, agrees with John's Gospel, and that it speaks of hanging in place of crucifixion. But he judges the forty-day herald a story with a purpose that is hard to think historical, the opposite of the Gospels' hurried trial before the Roman governor.
The Qur'an's answer
The Qur'an speaks of this charge and calls it false: those who broke the covenant spoke "against Mary a grave false charge" (4:156). Al-Tabari, an early commentator on the Qur'an (d. 923), explains the charge. It accused her of adultery, without proof, when she was innocent. The angels tell Mary that God (﷾) has chosen her and purified her, and chosen her above the women of all nations (3:42). The Qur'an calls her the woman "who guarded her chastity" (66:12). When she brought her child to her people, he spoke from the cradle. His first words were "I am indeed a servant of Allah", and he calls himself "kind to my mother" (19:27-33). Her people called her "sister of Aaron". The Prophet (ﷺ) explained that people named their children after earlier prophets and righteous people. So "sister of Aaron" can mean she had a brother named after the prophet Aaron. It does not confuse Mary with Miriam, the sister of Aaron. The page on that verse gives the full answer. His birth without a father is like Adam's: "He created him from dust" (3:59). Islam blames no one living today for an old charge: "no bearer of burdens can bear the burden of another" (6:164).
“(They have incurred divine displeasure): In that they broke their covenant; that they rejected the signs of Allah; that they slew the Messengers in defiance of right; that they said, "Our hearts are the wrappings (which preserve Allah's Word; We need no more)";- Nay, Allah hath set the seal on their hearts for their blasphemy, and little is it they believe;-”
“That they rejected Faith; that they uttered against Mary a grave false charge;”
Explains "their saying against Mary a grave false charge" (buhtanan 'aziman) as the slander of those whom the preceding verses describe: they accused her of zina, adultery, without proof or evidence, while she was innocent of it. He cites Ibn Abbas, al-Suddi and Juwaybir to the same effect.
“Behold! the angels said: "O Mary! Allah hath chosen thee and purified thee- chosen thee above the women of all nations.”
“And Mary the daughter of 'Imran, who guarded her chastity; and We breathed into (her body) of Our spirit; and she testified to the truth of the words of her Lord and of His Revelations, and was one of the devout (servants).”
“At length she brought the (babe) to her people, carrying him (in her arms). They said: "O Mary! truly an amazing thing hast thou brought! "O sister of Aaron! Thy father was not a man of evil, nor thy mother a woman unchaste!" But she pointed to the babe. They said: "How can we talk to one who is a child in the cradle?" He said: "I am indeed a servant of Allah: He hath given me revelation and made me a prophet; "And He hath made me blessed wheresoever I be, and hath enjoined on me Prayer and Charity as long as I live; "(He) hath made me kind to my mother, and not overbearing or miserable; "So peace is on me the day I was born, the day that I die, and the day that I shall be raised up to life (again)"!”
Al-Mughira ibn Shu'ba says the Christians of Najran asked him how the Qur'an could call Mary "sister of Aaron" when Moses lived long before Jesus. He put the question to the Prophet (ﷺ), who answered that people used to name their children after the prophets and righteous people who had gone before them.
“The similitude of Jesus before Allah is as that of Adam; He created him from dust, then said to him: "Be". And he was.”
“Say: "Shall I seek for (my) Cherisher other than Allah, when He is the Cherisher of all things (that exist)? Every soul draws the meed of its acts on none but itself: no bearer of burdens can bear the burden of another. Your goal in the end is towards Allah: He will tell you the truth of the things wherein ye disputed."”
Perspectives
Jewish (the rabbinic tradition)
The lines about Ben Stada's family are aggadah, story material set inside a legal discussion, not law. The Sages called Ben Stada a fool, and Rabbenu Tam denied that he was Jesus.
The lines about Ben Stada's mother are aggadah, a story inside a legal discussion, not law. The Sages call Ben Stada a fool.
Rebuttal
A story is not law, but the charge it records is old. Celsus' Jewish speaker made it about Jesus by name, about 175-180 CE. The Qur'an calls it "a grave false charge" (4:156).
The Tosafot are medieval notes on the Talmud. In them Rabbenu Tam says Ben Stada is not Yeshu ha-Notzri, because the two lived more than a century apart.
Rebuttal
That answers the name Ben Stada only. The Talmud itself says Ben Stada was also called Ben Pandira. The Tosefta ties the name Pantera to Jesus, and Celsus' Jewish speaker names Panthera as his father.
Academic study of the passages
Schäfer reads the Babylonian passage as a late counter-story to the Gospels' birth accounts. Herford saw in it old taunts against Jesus whose meaning had been lost. Others, Klausner among them, held that Ben Stada was another man. None of them treats it as history.
The Talmud's passage and Celsus most likely draw on common Jewish sources, and answer the Gospels' birth stories.
Rebuttal
If so, the story answers the Gospels and is not independent history. None of these scholars treats it as history.
The name may have been attached to Jesus late, or may belong to another man.
Rebuttal
Even on this view, Celsus' Jewish speaker made the charge about Jesus by name. Schäfer judges another Pantera unlikely, since the name is rare in Hebrew and Aramaic.
Mainstream Christian
Mary conceived as a virgin by the Holy Spirit, as Matthew and Luke record. The charge of adultery is a slander against that birth. Luke ties the birth to his being called the Son of God.
Matthew says Mary was found with child of the Holy Ghost before she and Joseph came together.
Rebuttal
The Qur'an agrees that Mary conceived without a man. A birth without a father shows God's (﷾) power, as with Adam; it does not make the child God.
Luke has Mary say she knows no man, and the angel call her child the Son of God.
Rebuttal
Luke's own genealogy calls Adam "the son of God", and no Christian makes Adam divine. Jesus is like Adam, created when God (﷾) said "Be".
Where the academy stands
Status: Majority view
Most modern scholars of these passages (Herford, Schäfer and others) hold that the Babylonian Talmud's Yeshu ha-Notzri passages refer to Jesus of Nazareth. They show him as the rabbis imagined him. A minority, notably Johann Maier, argued that the identification is late. Whether every Ben Stada or Ben Pandira text meant Jesus is more disputed. Rabbenu Tam denied it, and Joseph Klausner held that Ben Stada was another man. None of these scholars treats the stories as history. On the name Pantera, Herford holds that the Tosefta attaches it to Jesus, and Schäfer adds the Jewish speaker in Celsus.
The Islamic perspective
Mary was chaste, chosen and purified. The charge against her is a grave falsehood. Jesus was born without a father by God's (﷾) command, as Adam was created without father or mother. He was a servant and messenger of God (﷾), not God.
The Qur'an calls the charge against Mary "a grave false charge".
God (﷾) chose Mary and purified her, and chose her above the women of all nations.
Jesus was created by God's (﷾) command, as Adam was created from dust.
The infant Jesus defended his mother and called himself a servant of Allah (﷾).