LYNN – Lizette Milian spent Mother’s Day driving her two grandchildren to visit their mother in a Connecticut prison, while other mothers opened candy boxes and greeting cards on Sunday.”I took custody because there was no one else,” Milian said during a recent interview.Xavier, 9, and Isaac, 4, have been living with Milian since their mother was arrested. Milian’s son is Xavier’s father, but Isaac is not Milian’s biological relative.The number of grandparents like Milian raising grandchildren is growing, said Lynn Community Connections Coalition Director Pamela Freeman.In 2002, Freeman said Connections, a non-profit organization working with other groups to help strengthen families, conducted a survey identifying 2,900 Lynn-area grandparents raising their children’s children.”I think the number has to have tripled,” Freeman said.Freeman said the “primary reason” children like Isaac and Xavier end up with their grandparents is substance abuse and the damage it does.”Children of the grandparents are not able to care for their children,” she said.Mary Johnson was 52 when she brought her newborn grandson home from the hospital 18 years ago.Her son, the boy’s father, was not around to care for Devorn and, Johnson said, the child’s mother couldn’t take care of the baby, Johnson said.Devorn’s brother, Charles, 1, also moved into Johnson’s, and her late husband Calvin’s, home near Sluice Pond.For Mary Johnson, a career General Electric employee, raising two babies was not her first experience mothering grandchildren.A fractured relationship 25 years ago between one of her son’s and the mother of his children left the woman with custody of two boys and a girl.Johnson moved the three children into her home after learning they were referred to the state Department of Social Services and sent to a foster home when their mother left them with a baby sitter.”They each had one pair of socks and one pair of underwear,” she said.Johnson said she was fortunate her 10 children could pitch in and help raise Robert, then 6, Toya and Curtis.Johnson’s older children had helped out with their younger siblings when she worked a night shift at the River Works, but she said raising grandchildren in her mid-40s meant she had to start shuttling the children to doctor’s appointments and teacher conferences.”I had to get used to it again,” she said.For Milian, raising Isaac and Xavier meant moving out of her one-bedroom apartment and putting her active church life on the back burner to arrange schools and after-school programs for the boys.Xavier attends the Shoemaker School and goes to the YMCA after school and Isaac spends his afternoons at the Gregg House off Broad Street after mornings at the Harrington School. Milian said the biggest initial challenge she faced as a grandparent raising children was navigating the maze of agencies recommended to her for assistance.After one frustrating experience in a state office, Milian said she took a seat in the agency’s waiting room and announced: “I’m sitting here and not moving.”Her persistence secured her a spot for Isaac in Gregg House’s after-school program, but she said she still seeking additional help for his attention deficit problems.”Isaac has been a challenge,” she said.Instead of planning her retirement, Milian is stretching her income to cover household expenses.”Two kids eat like you wouldn’t believe,” she said.Freeman said grandparents raising grandchildren need help obtaining additional money to meet expenses.”You are talking about people who can barely push their financial envelope, who are living on fixed incomes,” she said.Freeman said Connections is looking for ways to take a cue from Boston and bring public housing dedicated to grandparents raising grandchildren to Lynn. Although she is raising her grandchildren in her own home, Johnson thinks grandparent-dedicated housing is a great idea.”It would be good because a lot of grandparents get their grandchildren and have to get a bigger place,”