Track fine-tune experiments & datasets with Weights & Biases

API
1

If you want to monitor and compare your different fine-tunes, you can now sync them to Weights & Biases with one command:

openai wandb sync

Make sure to upgrade openai first with pip install --upgrade openai.

image
image1093Ă—488 72.2 KB

You may need to install wandb first with pip install wandb.

It will let you visualize and compare your models.

image
image1600Ă—779 106 KB

You can also explore your datasets and check your model predictions.

image
image1098Ă—775 75.9 KB

Finally it let you build cool interactive reports to document your results.

You can find some example on the documentation page.

6 Likes
2

This looks amazing! I’ve never seen wandb before. Strange though - when I try to openai wandb sync, it shows that I’m logged inmshows my organization, but say s file object is missing with a 404 - and the filename isn’t one of the files I’ve ever uploaded (as far as I can tell).

When i run openai api files.list it’s not a file in that list.

Error: No such File object: file-6qTsDlhcJzCZd6of3xfEPOFa (HTTP status code: 404)

1 Like
3

The file should be a dataset you have used but may have a different name. Can you make sure you use the right API key?

Also for uploading your files you may need to have a credit card on file with your OpenAI account for security purposes. But even without it your runs should be synced and you should be able to visualize them.

4

I did a api fine_tuned.list query to see the files it returns. That one wants one of them. OpenAI is certainly has my cc on file based on my usage last month :joy::joy::joy:

I’ll play with it some other day when I understand more.

2 Likes
5

I would assume it’s one of your training or validation files. You can try to retrieve it manually.

Did the training metrics upload properly to your W&B dashboard?

1 Like
6

Unfortunately have this error:

$ openai wandb sync
wandb: Currently logged in as: louis030195 (use `wandb login --relogin` to force relogin)
Traceback (most recent call last):
  File "/home/foo/gpu/env/bin/openai", line 8, in <module>
    sys.exit(main())
  File "/home/foo/gpu/env/lib/python3.8/site-packages/openai/_openai_scripts.py", line 63, in main
    args.func(args)
  File "/home/foo/gpu/env/lib/python3.8/site-packages/openai/cli.py", line 542, in sync
    resp = openai.wandb_logger.WandbLogger.sync(
  File "/home/foo/gpu/env/lib/python3.8/site-packages/openai/wandb_logger.py", line 73, in sync
    fine_tune_logged = [
  File "/home/foo/gpu/env/lib/python3.8/site-packages/openai/wandb_logger.py", line 74, in <listcomp>
    cls._log_fine_tune(
  File "/home/foo/gpu/env/lib/python3.8/site-packages/openai/wandb_logger.py", line 138, in _log_fine_tune
    results_id = fine_tune["result_files"][0]["id"]
IndexError: list index out of range
7

Interesting, it’s most likely due to a failed or cancelled run.
You can try openai wandb sync -n 5 to log the last 5 fine-tunes to see if that works.

I may have to do something specific for cancelled or failed runs.

8

Does this only work with fine tunes or can I use it on a classification model as well? If so, any advice on how to do it?

9

@louis030195 , I believe the error was due to a failed run.

Can you confirm this branch fixes your error:
pip install --upgrade git+https://github.com/openai/openai-python.git@refs/pull/74/merge

10

@curranjanssens Yes you can use it to log your predictions for example.

You just create a table that could contain for example “prompt”, “completion” & “target” and then log it as a wandb Table. See guide.

11

@louis030195 Just wondering if you had a chance to test the fix?
Let me know if it works for you so I make a formal PR to integrate it into the master branch.

12

yes it worked thanks :slight_smile:

1 Like