How to calculate the tokens when using function call

API
10

Thanks a lot. I’ve try and compare with the return from API. Then I got below.


    public static int tokens(String modelName, Object functionCall, List<Function> functions) {
        Encoding encoding = getEncoding(modelName);
        int sum = 0;
        if (Preconditions.isNotBlank(functionCall)) {
            if (functionCall instanceof JSONObject) {
                sum += tokens(encoding, functionCall.toString());
            }
        }
        for (Function function : functions) {
            sum += tokens(encoding, function.getName());
            sum += tokens(encoding, function.getDescription());
            if (Preconditions.isNotBlank(function.getParameters())) {
                JSONObject jsonObject = (JSONObject) function.getParameters();
                if (jsonObject.containsKey("properties")) {
                    for (String propertiesKey : jsonObject.getJSONObject("properties").keySet()) {
                        sum += tokens(encoding, propertiesKey);
                        JSONObject v = jsonObject.getJSONObject("properties").getJSONObject(propertiesKey);
                        for (String field : v.keySet()) {
                            if ("type".equals(field)) {
                                sum += 2;
                                sum += tokens(encoding, v.getString("type"));
                            } else if ("description".equals(field)) {
                                sum += 2;
                                sum += tokens(encoding, v.getString("description"));
                            } else if ("enum".equals(field)) {
                                sum -= 3;
                                for (Object o : v.getJSONArray(field)) {
                                    sum += 3;
                                    sum += tokens(encoding, o.toString());
                                }
                            } else {
                                log.warn("not supported field {}", field);
                            }
                        }
                    }
                }
                sum += 11;
            }
        }
        sum += 12;
        return sum;
    }

It is a function in file TikTokenUtils and it cover type/description/enum properties. And it is just matched with the usage returned from API.

Let me know if you have any better way.

2 Likes