Average GPT-4 rumor in 2023.
On a serious note tho, GPT 4 isn't probably going to be too big compared to GPT-3 as we kind of already have hit the limits of computations. However here are some of the things to expect (this is just my personal opinion based on my knowledge of the field):
- the size won't be bigger than ~10x at best or more likely it will be just the same size.
- it may include some optimization tricks from recent research. like longer context, better layers, faster attention, etc.
- the quality of the data would probably be the biggest factor, there is a possibility they might be training the new model on YouTube video transcripts.
- multi-modality could be a possibility like GPT directly outputting images along with the text but i see it less probable. More probable is for them to use dalle to synthesize images.
- there will probably be more specialized versions/experts like chatgpt, along with the core special one i.e. davinci.
- it will more likely have longer context (currently it has 8k tokens), it might have some sort of memory to increase context way beyond 50k.
