Results of NeFL with five submodels and SOTA baselines on five datasets under IID settings. We report test performance including Top-1 classification accuracies (%) for the worst-case submodel and the average performance over five submodels.
| Model | Method | CIFAR-10 | CIFAR-100 | CINIC-10 | SVHN | ||||
|---|---|---|---|---|---|---|---|---|---|
| Worst | Avg | Worst | Avg | Worst | Avg | Worst | Avg | ||
| ResNet18 | HeteroFL | 80.62 | 84.26 | 41.33 | 47.09 | 67.55 | 70.40 | 91.82 | 93.46 |
| FjORD | 85.12 | 87.32 | 49.29 | 52.67 | 71.95 | 74.98 | 94.31 | 93.97 | |
| DepthFL | 64.80 | 82.44 | 31.68 | 49.56 | 54.51 | 71.42 | 91.54 | 93.97 | |
| ScaleFL | 79.47 | 85.18 | 41.00 | 49.76 | 70.55 | 73.85 | 93.15 | 94.53 | |
| NeFL(ours) | 87.71 | 89.02 | 55.22 | 56.26 | 75.02 | 76.68 | 94.72 | 95.22 | |
| ResNet34 | HeteroFL | 79.51 | 83.16 | 34.96 | 39.75 | 67.39 | 69.62 | 89.86 | 92.39 |
| FjORD | 85.12 | 87.36 | 47.59 | 50.7 | 71.58 | 74.19 | 93.83 | 94.63 | |
| DepthFL | 25.73 | 75.30 | 14.51 | 46.79 | 32.05 | 67.04 | 74.33 | 89.96 | |
| ScaleFL | 54.72 | 81.05 | 22.62 | 46.41 | 49.69 | 69.43 | 86.46 | 93.21 | |
| NeFL(ours) | 87.71 | 89.02 | 55.22 | 56.26 | 75.02 | 76.68 | 94.72 | 95.22 | |
Results of NeFL utilizing pre-trained models as initial weights for CIFAR-10 dataset under IID (left) and non-IID (right) settings. Numbers in parentheses denote the performance difference compared to scratch, with blue indicating improvement from pretraining, and red indicating degradation.
| Model | Method | IID | non-IID | ||
|---|---|---|---|---|---|
| Worst | Avg | Worst | Avg | ||
| Pre-trained ResNet18 | HeteroFL | 78.26(↓2.36) | 84.48(↑0.22) | 71.95(↓4.30) | 76.17(↓3.94) |
| FjORD | 86.37(↑1.25) | 88.91(↑1.59) | 81.81(↑6.00) | 81.96(↑3.97) | |
| DepthFL | 47.76(↓17.04) | 82.86(↑0.42) | 39.78(↓19.83) | 67.71(↓9.18) | |
| ScaleFL | 79.34(↓0.13) | 86.16(↑0.98) | 69.47(↑6.00) | 78.01(↓0.48) | |
| NeFL(ours) | 88.61(↑1.75) | 89.60(↑1.72) | 82.91(↑1.65) | 85.85(↑4.14) | |
| Pre-trained ResNet34 | HeteroFL | 79.97(↑0.46) | 84.34(↑1.18) | 72.33(↓3.70) | 78.20(↓1.43) |
| FjORD | 87.08(↑1.96) | 89.37(↑2.01) | 78.20(↑3.50) | 78.90(↑2.89) | |
| DepthFL | 52.08(↑26.35) | 83.63(↑8.33) | 42.09(↑11.67) | 79.86(↑9.10) | |
| ScaleFL | 67.66(↑12.94) | 85.77(↑4.72) | 52.59(↑20.25) | 78.29(↑5.89) | |
| NeFL(ours) | 88.36(↑0.65) | 91.14(↑2.12) | 83.62(↑2.86) | 86.48(↑3.18) | |