You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-18 - 16:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Wed Mar 18, 2026 12:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1049929342,45sleep20-21swapper/207:05:242
1130529279,9sleep00-21swapper/007:09:230
1101027966,9sleep30-21swapper/307:05:323
922727659,13sleep10-21swapper/107:05:101
92092530,0sleep049-21kcompactd009:12:230
11441992413,6cyclictest225-21systemd-journal10:41:332
1144199230,23cyclictest0-21swapper/211:55:482
1143599221,2cyclictest141rcu_preempt11:03:410
1143999212,2cyclictest4433-21cron11:45:001
1143999210,20cyclictest0-21swapper/108:25:181
1143599218,6cyclictest27411-21munin-run11:20:000
1143599218,6cyclictest27411-21munin-run11:20:000
1144599204,15cyclictest0-21swapper/311:14:123
11441992018,1cyclictest32-21ksoftirqd/209:55:322
11441992011,8cyclictest32-21ksoftirqd/210:39:152
1144199200,18cyclictest0-21swapper/211:06:582
11439992013,4cyclictest8499-21sh11:27:281
1144599190,18cyclictest0-21swapper/309:31:263
1143999190,18cyclictest0-21swapper/109:34:341
1143599190,3cyclictest0-21swapper/012:31:150
1143599190,3cyclictest0-21swapper/010:03:240
11441991816,1cyclictest32-21ksoftirqd/209:30:112
11441991813,2cyclictest32-21ksoftirqd/210:59:572
1143999182,1cyclictest4241-21diskmemload11:10:301
1143999181,3cyclictest28253-21rm09:43:151
1143999180,15cyclictest0-21swapper/109:50:201
1143599186,6cyclictest3442-21munin-run11:25:000
1143599180,2cyclictest0-21swapper/010:16:310
11441991714,2cyclictest32-21ksoftirqd/212:17:352
1144199170,15cyclictest0-21swapper/211:28:552
1143999172,1cyclictest0-21swapper/111:15:531
1143999172,1cyclictest0-21swapper/111:15:531
1143999170,16cyclictest0-21swapper/110:40:131
1143999170,16cyclictest0-21swapper/110:33:021
1143599177,5cyclictest7675-21dpkg10:10:000
1144599163,6cyclictest0-21swapper/311:55:123
11445991615,1cyclictest0-21swapper/310:27:413
1144599161,14cyclictest0-21swapper/312:24:593
1144599160,15cyclictest0-21swapper/308:52:033
11441991615,1cyclictest32-21ksoftirqd/211:40:012
1144199160,16cyclictest0-21swapper/210:45:042
1143999162,1cyclictest7772-21ssh10:48:261
11439991615,1cyclictest0-21swapper/109:57:351
1143999160,16cyclictest0-21swapper/110:16:311
87502152,1sleep04241-21diskmemload09:31:040
1144599150,1cyclictest0-21swapper/309:35:343
1144599150,15cyclictest0-21swapper/312:34:143
1144599150,14cyclictest4111-21kworker/3:108:47:093
1144599150,14cyclictest4111-21kworker/3:108:42:553
1144199154,7cyclictest32-21ksoftirqd/209:10:222
1144199153,2cyclictest141rcu_preempt10:51:472
1144199152,6cyclictest29477-21basename09:25:002
11441991513,1cyclictest32-21ksoftirqd/209:36:192
1144199150,14cyclictest0-21swapper/211:10:122
1144199150,14cyclictest0-21swapper/210:18:232
1143999159,5cyclictest19592-21sed07:25:161
11439991514,1cyclictest25-21ksoftirqd/109:11:291
1143999150,14cyclictest0-21swapper/111:46:261
87902140,0sleep30-21swapper/312:06:093
1144599141,5cyclictest0-21swapper/310:35:003
11445991411,2cyclictest19701-21wc10:55:183
1144599140,13cyclictest25021-21wc10:00:193
1144199140,2cyclictest141rcu_preempt10:20:252
1144199140,1cyclictest141rcu_preempt11:15:482
1144199140,1cyclictest141rcu_preempt11:15:482
1144199140,14cyclictest0-21swapper/212:12:332
1144199140,13cyclictest28217-21grep11:00:192
1144199140,12cyclictest0-21swapper/209:49:582
1143999148,5cyclictest2027-21apache211:56:041
11439991410,3cyclictest31622-21basename09:00:001
11439991410,3cyclictest15143-21cut07:15:211
1143999140,2cyclictest0-21swapper/111:35:061
11435991412,1cyclictest18719-21kworker/u8:109:55:120
11435991410,2cyclictest20782-21grep11:35:140
1143599140,8cyclictest29384-21tr12:00:000
1143599140,12cyclictest8660-21grep10:10:140
47182131,1sleep04739-21ssh12:04:310
206132130,0chrt0-21swapper/009:19:360
1144599139,3cyclictest9243-21cut10:10:173
1144599139,3cyclictest526-21grep10:05:163
1144599139,3cyclictest12889-21cat11:50:003
1144599139,2cyclictest30595-21grep08:55:193
1144599138,3cyclictest10183-21cron08:15:003
1144599132,4cyclictest31543-21apt-get12:40:003
1144599131,3cyclictest21091-21ssh11:55:003
11445991311,1cyclictest4241-21diskmemload11:37:103
11445991310,2cyclictest27243-21taskset10:40:193
11445991310,2cyclictest0-21swapper/311:40:143
1144599130,4cyclictest0-21swapper/310:48:043
1144599130,12cyclictest15258-21head09:15:263
1144599130,12cyclictest0-21swapper/310:50:023
1144199139,3cyclictest3498-21cat11:25:002
1144199139,3cyclictest31355-21grep12:00:222
1144199139,3cyclictest12893-21cut08:20:132
1144199138,5cyclictest2035-21apache212:30:412
1144199138,4cyclictest12551-21expr11:30:142
1144199137,2cyclictest32-21ksoftirqd/210:01:002
1144199134,7cyclictest19636-21runrttasks09:18:502
1144199133,9cyclictest30032-21grep09:25:132
1144199133,6cyclictest0-21swapper/212:35:132
11441991310,2cyclictest9311-21grep08:10:212
11441991310,2cyclictest32174-21tr09:00:132
1144199130,1cyclictest0-21swapper/210:07:072
1144199130,12cyclictest5042-21grep08:00:262
1144199130,12cyclictest2895-21grep09:05:162
1143999139,3cyclictest19709-21cron08:34:591
11439991312,1cyclictest0-21swapper/108:40:111
11439991311,1cyclictest4241-21diskmemload10:51:241
11439991310,2cyclictest31411-21grep07:50:161
11439991310,2cyclictest31411-21grep07:50:161
11439991310,2cyclictest16068-21grep12:10:251
11439991310,2cyclictest1104-21ls10:05:201
11439991310,2cyclictest10348-21taskset11:09:421
1143999130,1cyclictest0-21swapper/112:29:031
1143999130,1cyclictest0-21swapper/110:00:021
1143999130,13cyclictest0-21swapper/110:23:031
1143999130,11cyclictest0-21swapper/112:15:131
1143599139,3cyclictest12408-21grep07:10:170
1143599138,3cyclictest17896-21tr08:30:130
11435991310,2cyclictest21946-21grep09:20:150
11435991310,2cyclictest19748-21grep10:35:250
11435991310,2cyclictest13723-21cat12:10:000
1144599129,2cyclictest8409-21tr12:25:203
1144599129,2cyclictest6306-21cut08:05:153
1144599129,2cyclictest5882-21wc09:10:183
1144599129,2cyclictest4208-21grep11:25:133
1144599129,2cyclictest39-21ksoftirqd/311:00:143
1144599129,2cyclictest31741-21grep09:45:163
1144599129,2cyclictest29862-21grep11:20:253
1144599129,2cyclictest28561-21tr07:45:143
1144599129,2cyclictest21087-21cut08:35:183
1144599129,2cyclictest20306-21cut11:15:163
1144599129,2cyclictest20306-21cut11:15:163
1144599129,2cyclictest16101-21grep12:10:263
1144599128,2cyclictest347-21cron07:55:003
1144599128,2cyclictest347-21cron07:55:003
1144599121,3cyclictest30330-21munin-run12:20:003
1144599121,2cyclictest10718-21cron11:10:003
1144599120,12cyclictest0-21swapper/309:22:243
1144599120,11cyclictest0-21swapper/308:30:123
1144199129,2cyclictest9214-21sed10:10:172
1144199129,2cyclictest5812-21sed11:45:172
1144199129,2cyclictest22086-21sh08:40:002
1144199129,2cyclictest1648-21cut07:55:142
1144199129,2cyclictest15883-21cut08:25:162
1144199128,3cyclictest30443-21sh07:50:012
1144199122,10cyclictest32-21ksoftirqd/212:20:132
11441991210,1cyclictest7528-21df12:25:132
1143999129,2cyclictest9158-21ssh10:10:171
1143999129,2cyclictest28747-21grep08:50:261
1143999129,2cyclictest23846-21tr07:35:121
1143999129,2cyclictest22083-21cron08:40:001
1143999129,2cyclictest13495-21grep11:30:191
1143999129,2cyclictest12444-21head07:10:171
1143999128,3cyclictest21476-21ssh09:20:051
1143999126,5cyclictest28062-21cron07:45:001
11439991211,1cyclictest38150irq/131-snd_hda12:36:531
11439991211,1cyclictest0-21swapper/112:07:431
11439991210,1cyclictest3834-21grep09:05:251
1143999120,2cyclictest21773-21rm10:56:201
1143599129,2cyclictest32488-21grep09:00:150
1143599129,2cyclictest30217-21tail08:55:160
1143599129,2cyclictest26836-21tail10:40:160
1143599129,2cyclictest24060-21fgrep12:15:230
1143599129,2cyclictest22157-21apt09:40:000
1143599129,2cyclictest21037-21grep11:55:000
1143599129,2cyclictest16299-21ssh11:32:010
1143599129,2cyclictest12736-21ls11:50:000
1143599129,2cyclictest12129-21ssh11:10:170
1143599125,2cyclictest9422-21apt-get10:30:120
1143599120,1cyclictest0-21swapper/011:05:080
1144599118,2cyclictest540-21ssh12:01:173
1144599118,2cyclictest475-21in:imuxsock07:20:003
1144599118,2cyclictest3599-21grep09:05:253
1144599118,2cyclictest3187-21date08:00:003
1144599118,2cyclictest31254-21rm09:25:203
1144599118,2cyclictest26534-21sed07:40:143
1144599118,2cyclictest26454-21head10:20:193
1144599118,2cyclictest22489-21ssh09:40:053
1144599118,2cyclictest19637-21grep10:35:253
1144599118,2cyclictest18894-21cut07:25:133
1144599118,2cyclictest16273-21grep09:55:163
1144599118,2cyclictest13855-21wc07:15:003
1144599118,2cyclictest133392sleep308:20:133
1144599115,3cyclictest10460-21apt-get08:15:133
1144599110,10cyclictest8369-21grep09:50:193
1144199119,1cyclictest14840-21expr07:15:182
1144199118,2cyclictest9403-21ssh10:30:112
1144199118,2cyclictest836-21sh11:42:142
1144199118,2cyclictest6462-21sed08:05:152
1144199118,2cyclictest5961-21cut12:05:132
1144199118,2cyclictest28183-21wc07:45:002
1144199118,2cyclictest21875-21sort07:30:172
1144199118,2cyclictest19728-21ssh11:53:592
1144199118,2cyclictest19673-21chrt08:34:352
1144199118,2cyclictest17674-21grep07:20:242
1144199118,2cyclictest14130-21ssh09:54:212
1144199118,1cyclictest311rcuc/207:10:182
1144199115,3cyclictest32-21ksoftirqd/209:40:152
1144199113,5cyclictest20922-21cron07:30:002
1144199110,10cyclictest0-21swapper/210:29:002
1143999119,1cyclictest28595-21grep07:45:141
1143999118,2cyclictest7933-21ssh10:29:131
1143999118,2cyclictest6363-21cut08:05:151
1143999118,2cyclictest30338-21ssh11:01:301
1143999118,2cyclictest27795-21ssh11:20:071
1143999118,2cyclictest16116-21ssh11:51:091
1143999118,2cyclictest14798-21cat09:35:151
1143999117,3cyclictest18041-21grep10:35:131
1143999115,6cyclictest38150irq/131-snd_hda12:31:471
1143999114,4cyclictest225-21systemd-journal07:25:001
1143999113,4cyclictest225-21systemd-journal12:20:001
1143999110,2cyclictest0-21swapper/107:55:111
1143999110,10cyclictest13055-21sh09:15:031
1143999110,10cyclictest0-21swapper/109:26:211
1143999110,0cyclictest0-21swapper/109:45:371
1143599118,2cyclictest9172-21ssh09:50:260
1143599118,2cyclictest7089-21taskset10:47:530
1143599118,2cyclictest6680-21ssh11:26:010
1143599118,2cyclictest32095-21sensors_fan07:50:220
1143599118,2cyclictest32095-21sensors_fan07:50:220
1143599118,2cyclictest30362-21chrt07:49:050
1143599118,2cyclictest29777-21ssh09:25:050
1143599118,2cyclictest29765-21ssh10:22:270
1143599118,2cyclictest24896-21egrep08:45:140
1143599118,2cyclictest24741-21ssh09:40:290
1143599118,2cyclictest23709-21ssh12:35:120
1143599118,2cyclictest21654-21ssh10:56:140
1143599118,2cyclictest20938-21cat08:35:170
1143599118,2cyclictest18028-21ls07:20:280
1143599118,2cyclictest16309-21ssh12:10:310
1143599118,2cyclictest15077-21wc08:25:010
1143599118,2cyclictest14192-21cut07:15:160
1143599118,2cyclictest11201-21ssh10:50:180
1143599118,2cyclictest11194-21ssh12:27:080
1143599118,2cyclictest10705-21cut08:15:140
1143599117,3cyclictest3367-21grep08:00:130
1143599117,3cyclictest29117-21grep11:40:130
1143599111,8cyclictest27117-21apache_accesses08:50:110
1143599110,10cyclictest893-21cut07:55:110
1143599110,10cyclictest30712-21ssh12:20:080
1144599108,1cyclictest5188-21chrt08:01:543
1144599108,1cyclictest15888-21grep08:25:163
1144599107,2cyclictest32118-21tr09:00:133
1144599107,2cyclictest21487-21grep07:30:153
1144599107,2cyclictest17267-21rm10:15:183
1144599106,3cyclictest25265-21grep07:35:243
1144599100,9cyclictest17137-21cut07:20:193
1144599100,9cyclictest16188-21sh11:31:563
1144199108,1cyclictest32745-21taskset07:54:122
1144199108,1cyclictest32745-21taskset07:54:122
1144199108,1cyclictest30937-21wc08:55:252
1144199108,1cyclictest27327-21cat08:50:142
1144199108,1cyclictest265312sleep208:45:572
1144199108,1cyclictest253432sleep207:35:492
1144199108,1cyclictest11055-21cut08:15:162
1144199102,2cyclictest141rcu_preempt08:40:152
1143999108,2cyclictest12260-21seq08:16:101
1143999108,1cyclictest3649-21cat08:00:141
1143999108,1cyclictest247472sleep108:45:111
1143999107,2cyclictest8788-21grep08:10:171
1143999107,2cyclictest31734-21grep12:00:261
1143999100,9cyclictest13567-21expr08:20:151
1143599108,1cyclictest5838-21cut08:05:120
1143599108,1cyclictest31993-21head09:45:170
1143599108,1cyclictest27493-21cat07:40:240
1143599108,1cyclictest23774-21cat07:35:120
1143599108,1cyclictest19061-21cut07:25:140
1143599108,1cyclictest1104-21ssh10:25:130
1143599107,2cyclictest8818-21grep08:10:170
1143599107,2cyclictest15999-21tail08:25:170
1143599100,9cyclictest23323-21basename07:35:000
114399997,2cyclictest31889-21apache_volume09:00:111
114399992,6cyclictest21427-21tr07:30:151
114359998,1cyclictest23567-21grep08:40:190
114359996,2cyclictest2122-21apt09:05:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional