Hello,
I am trying to finetune a model to act like a country marketing specialist. It has been very tough as I got so many bad models. Today, I increased the epochs from 4 to 16 and I got fairly good output with some unexpected outputs so it still random and not being able to generalize. I followed the data designing format, - adding space at the begging of the completion, adding a stop like +++ to the end of the prompt and same at the end of completing ###.
My data consist of 200+ prompts, and I firstly intended to give it some common knowledge about my country, and if it worked well, I start with marketing based
Here is some examples of my prompts:
{"prompt":"What are the traditional wedding customs in Iraq? ###","completion":" Traditional wedding customs in Iraq vary among different regions and communities. They often involve vibrant celebrations, music, dancing, and rituals such as the signing of the marriage contract, henna ceremonies, and festive feasts. END"}
{"prompt":"Describe the role of tribalism in Iraqi society. ###","completion":" Tribalism has long played a significant role in Iraqi society. Tribal affiliations and structures have influenced social, political, and economic dynamics, particularly in rural areas, where tribal customs and traditions are still prevalent. END"}
{"prompt":"Explain the significance of the Battle of Ctesiphon in Iraq's history. ###","completion":" The Battle of Ctesiphon was a major historical event in Iraq's history. It took place in 363 AD and resulted in the capture of the Sasanian capital by the Roman Empire, leading to significant shifts in regional power dynamics. END"}
{"prompt":"What are some famous archaeological sites in Iraq? ###","completion":" Iraq is home to several famous archaeological sites, such as the ancient city of Ur, the ruins of Nineveh, the ancient city of Nimrud, and the Ziggurat of Ur. These sites offer valuable insights into ancient civilizations. END"}
{"prompt":"Explain the traditional clothing worn in Iraq. ###","completion":" Traditional clothing in Iraq varies among different regions and communities. Examples include the dishdasha, abaya, and hijab for men and women, as well as the Kurdish traditional attire known as the chwarshah. END"}
{"prompt":"What are some famous literary works from Iraq? ###","completion":" Iraq has a rich literary heritage. Some famous literary works include 'The Thousand and One Nights,' a collection of folk tales, 'The Cairo Trilogy' by Naguib Mahfouz, and the poems of renowned Iraqi poet Badr Shakir al-Sayyab. END"}
{"prompt":"Explain the significance of the city of Najaf in Iraq. ###","completion":" Najaf is a city in Iraq with great religious and cultural importance. It is home to the Shrine of Imam Ali, one of the holiest sites in Shia Islam and a destination for pilgrims from around the world. END"}
{"prompt":"What are some traditional musical instruments in Iraq? ###","completion":" Traditional musical instruments in Iraq include the oud, qanun, santur, djoze, and tabla. These instruments are integral to Iraqi music, which encompasses various genres and styles influenced by the region's diverse cultural heritage. END"}
{"prompt":"Describe the Marsh Arabs and their unique way of life in Iraq. ###","completion":" The Marsh Arabs, also known as Ma'dan, have a distinct way of life in the marshlands of southern Iraq. They rely on fishing, agriculture, and the management of water resources to sustain their communities, maintaining a unique cultural identity. END"}
{"prompt":"What are the main challenges faced by Iraq in recent years? ###","completion":" Iraq has faced numerous challenges, including political instability, security issues, economic hardships, and the reconstruction and recovery from conflicts. Efforts are being made to address these challenges and promote stability and development. END"}
{"prompt":"What are the UNESCO World Heritage Sites in Iraq? ###","completion":" Iraq is home to several UNESCO World Heritage Sites, including the ancient city of Babylon, the Citadel of Erbil, the archaeological site of Hatra, and the marshlands of southern Iraq, known as the Ahwar of Southern Iraq. END"}
{"prompt":"Explain the cultural festival of Nowruz celebrated in Iraq. ###","completion":" Nowruz is a cultural festival celebrated in Iraq and other countries with Persian heritage. It marks the beginning of spring and involves various traditions, such as setting up the Haft Seen table and engaging in festive activities. END"}
{"prompt":"What are the main environmental challenges faced by Iraq? ###","completion":" Iraq faces various environmental challenges, including water scarcity, desertification, pollution, and the degradation of ecosystems. Efforts are being made to address these challenges through sustainable practices, conservation initiatives, and environmental regulations. END"}
And this is the command I ran to finetune my model:
openai api fine_tunes.create -t YKN.jsonl -m davinci --n_epochs 15
This is how I set the response :
response = openai.Completion.create(
model=model_id,
prompt=prompt,
max_tokens=300,
stop=' END',
temperature=0.5
)
Example of the output I am getting:
p: Hello?
C: ”
“Hey, it’s me.”
“Hey, yourself. What’s up?”
“I just wanted to call and say hi.”
“That’s so sweet. I miss you, too.”
“I miss you, too.”
“I miss you, too.”
“I miss you, too.”
“I miss you, too.”
“I miss you, too.”
“I miss you, too.”
“I miss you, too.”
“I miss you, too.”
“I miss you, too.”
“I miss you, too.”
“I miss you, too.”
“I miss you, too.”
“I miss you, too.”
“I miss you, too.”
“I miss you, too.”
“I miss you, too.”
“I miss you, too.”
“I miss you, too.”
“I miss you, too.”
“I miss
My question is, how can I make it answer the bilingual model (Arabic, English)?
- How can I have it answer common knowledge questions, like 1 +1?
- Does my model has common knowledge or it’s limited to the data knowledge?
- How should I decide the epochs number needed minding the overfitting and underfitting?
- Can I make my data in two languages? or how can I make the model answer in two languages?