You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-28 - 20:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Tue Jul 28, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1312229971,9sleep00-21swapper/007:06:180
1312529480,10sleep30-21swapper/307:06:203
1330929178,9sleep20-21swapper/207:08:452
1317028966,6sleep10-21swapper/107:06:551
1350299331,2cyclictest2987-21dump-pmu-power10:45:011
1350299290,2cyclictest0-21swapper/111:58:481
1350299284,23cyclictest6084-21diskmemload10:20:561
1350299283,21cyclictest225-21systemd-journal08:25:001
1350299271,7cyclictest6084-21diskmemload11:26:591
1350299271,1cyclictest6084-21diskmemload11:45:261
1349999270,27cyclictest0-21swapper/012:05:430
1350299264,21cyclictest32373-21munin-run12:20:011
1350299260,2cyclictest0-21swapper/111:44:021
1350299260,25cyclictest0-21swapper/112:20:121
1350299260,25cyclictest0-21swapper/110:02:381
1350299260,25cyclictest0-21swapper/107:45:121
1350299260,1cyclictest0-21swapper/109:46:241
1349999260,24cyclictest0-21swapper/012:22:140
1350299251,23cyclictest14907-21ssh09:15:111
1350299250,24cyclictest0-21swapper/112:25:141
1350299250,24cyclictest0-21swapper/110:55:141
1350899241,22cyclictest0-21swapper/307:10:133
1350299243,19cyclictest6084-21diskmemload10:54:401
1350299241,1cyclictest6084-21diskmemload10:45:131
1350299240,2cyclictest6084-21diskmemload11:36:161
1350299240,1cyclictest0-21swapper/109:12:541
1350299235,8cyclictest0-21swapper/108:10:131
1350299235,17cyclictest6084-21diskmemload11:51:021
1350299232,20cyclictest17255-21needreboot09:35:221
1350299231,1cyclictest6084-21diskmemload09:28:191
1350299230,22cyclictest0-21swapper/112:35:151
1350299230,1cyclictest0-21swapper/112:33:191
1350299230,1cyclictest0-21swapper/112:11:421
1350299230,1cyclictest0-21swapper/109:33:251
1349999230,3cyclictest0-21swapper/011:18:500
1350299222,19cyclictest6084-21diskmemload10:17:551
1350299222,19cyclictest0-21swapper/109:55:231
1350299221,20cyclictest0-21swapper/108:50:141
1350299221,1cyclictest6084-21diskmemload11:06:441
1349999220,2cyclictest0-21swapper/011:49:190
1350599210,20cyclictest0-21swapper/210:18:022
1350299213,15cyclictest6084-21diskmemload10:39:371
1350299213,15cyclictest6084-21diskmemload10:39:361
13502992116,4cyclictest38150irq/131-snd_hda10:25:131
1350299211,17cyclictest2069-21kworker/1:111:30:211
1350299210,2cyclictest0-21swapper/109:43:021
1350299210,20cyclictest0-21swapper/109:52:151
1350299210,1cyclictest0-21swapper/112:05:051
1350299210,1cyclictest0-21swapper/111:24:561
13499992120,1cyclictest0-21swapper/012:33:580
1350299201,1cyclictest6084-21diskmemload10:34:371
1350299201,18cyclictest6084-21diskmemload10:10:061
1350299200,1cyclictest0-21swapper/111:14:351
1350299200,0cyclictest0-21swapper/112:03:131
13499992016,2cyclictest0-21swapper/010:08:210
123982200,0sleep20-21swapper/209:51:472
13508991918,1cyclictest0-21swapper/312:34:033
1350599190,1cyclictest0-21swapper/209:58:242
1350299192,16cyclictest0-21swapper/108:05:191
1350299191,1cyclictest13820-21grep08:15:281
1350299190,2cyclictest0-21swapper/111:00:131
1350299190,18cyclictest0-21swapper/111:17:391
1350899188,9cyclictest1142-21runrttasks09:16:563
1350599180,1cyclictest0-21swapper/212:38:122
1350299180,1cyclictest0-21swapper/110:06:071
1350299180,1cyclictest0-21swapper/109:05:181
1350299180,1cyclictest0-21swapper/107:40:141
1350299180,16cyclictest0-21swapper/109:21:581
1349999185,7cyclictest7336-21awk12:05:000
1349999181,11cyclictest0-21swapper/008:40:150
1349999180,2cyclictest0-21swapper/010:44:570
1349999180,1cyclictest0-21swapper/010:02:470
13508991715,1cyclictest39-21ksoftirqd/311:44:353
1350299171,4cyclictest1-21systemd07:30:481
1350299169,6cyclictest6841-21tail08:00:251
1350299169,6cyclictest18244-21systemctl07:20:011
1350299167,7cyclictest3924-21tr07:55:201
1350299161,1cyclictest0-21swapper/107:25:221
1350299160,1cyclictest0-21swapper/107:20:131
1350299160,15cyclictest0-21swapper/108:35:211
1349999160,16cyclictest0-21swapper/009:38:180
13505991514,1cyclictest0-21swapper/210:51:042
13505991511,3cyclictest55650irq/127-enp2s0-12:03:292
1350599150,3cyclictest141rcu_preempt09:43:362
1350299158,6cyclictest27738-21chrt07:39:231
1350299158,6cyclictest14711-21wc07:10:171
1349999151,7cyclictest20924-21munin-run11:14:590
1349999150,12cyclictest0-21swapper/007:15:140
1350899149,2cyclictest1142-21runrttasks10:34:233
1350899142,12cyclictest0-21swapper/311:56:283
13508991410,3cyclictest10612-21sed09:50:263
1350899140,4cyclictest0-21swapper/309:22:533
13505991411,2cyclictest17349-21grep08:25:152
13505991411,2cyclictest1063-21wc09:25:272
13505991410,3cyclictest9107-21grep12:05:212
13505991410,3cyclictest7279-21cron08:05:002
1350299142,11cyclictest0-21swapper/108:55:201
1350299140,2cyclictest0-21swapper/107:50:271
1349999141,7cyclictest5175-21apt-get11:24:590
13499991414,0cyclictest0-21swapper/010:18:270
13499991410,3cyclictest20046-21cat10:55:000
1349999140,1cyclictest0-21swapper/009:59:360
1350899133,6cyclictest23438-21kworker/u8:210:45:023
13508991312,1cyclictest0-21swapper/312:17:513
13508991312,1cyclictest0-21swapper/310:56:523
13508991311,2cyclictest39-21ksoftirqd/310:52:123
1350899131,11cyclictest22276-21python07:25:263
13508991310,2cyclictest8715-21sed11:45:263
13508991310,2cyclictest39-21ksoftirqd/310:15:133
13508991310,2cyclictest16243-21cut12:10:153
13508991310,2cyclictest14424-21ls08:20:003
1350899130,12cyclictest23940-21ls08:40:013
1350599139,3cyclictest5081-21uname09:05:212
1350599134,5cyclictest225-21systemd-journal08:35:012
13505991312,1cyclictest0-21swapper/209:37:062
13505991310,2cyclictest7888-21tr09:30:132
13505991310,2cyclictest6665-21cut11:25:182
13505991310,2cyclictest30583-21sed11:20:162
13505991310,2cyclictest29318-21grep07:40:212
13505991310,2cyclictest26201-21grep08:40:292
13505991310,2cyclictest18553-21rm10:34:582
13505991310,2cyclictest15728-21tail09:15:162
1350599130,12cyclictest4031-21cat10:25:232
1350599130,12cyclictest28923-21ls10:40:222
1350599130,12cyclictest0-21swapper/207:14:152
1350299138,5cyclictest3574-21chrt09:03:351
1350299137,5cyclictest21286-21chrt08:31:371
1350299131,1cyclictest225-21systemd-journal08:30:011
13502991310,2cyclictest27492-21cut08:45:181
13502991310,2cyclictest27492-21cut08:45:171
13499991310,2cyclictest4104-21wc10:25:250
13499991310,2cyclictest25051-21cat09:20:250
13499991310,2cyclictest23310-21grep11:35:180
13499991310,2cyclictest15324-21grep11:50:150
13499991310,2cyclictest1099-21sort09:45:160
1349999130,8cyclictest0-21swapper/011:40:160
1349999130,12cyclictest29091-21grep11:00:140
1349999130,11cyclictest15040-21grep09:15:140
1349999130,11cyclictest0-21swapper/011:57:290
1350899129,2cyclictest8303-21sed08:05:173
1350899129,2cyclictest3304-21sed09:00:293
1350899129,2cyclictest30060-21grep11:20:133
1350899129,2cyclictest20122-21cut08:30:163
1350899129,2cyclictest17297-21ntpq09:35:233
1350899129,2cyclictest17267-21tr09:55:133
1350899129,2cyclictest1396-21cat07:50:203
1350899129,2cyclictest13493-21cyclictest12:07:323
1350899129,2cyclictest13452-21cat11:10:163
1350899128,3cyclictest24862-21ls07:30:253
1350899123,7cyclictest31866-21/usr/sbin/munin12:00:263
13508991210,1cyclictest11437-21cat08:10:253
13508991210,1cyclictest100832sleep311:27:103
1350899120,11cyclictest8548-21sed09:10:283
1350599129,2cyclictest733-21grep12:20:142
1350599129,2cyclictest6328-21cron11:44:592
1350599129,2cyclictest4525-21sed10:45:152
1350599129,2cyclictest4043-21wc07:55:222
1350599129,2cyclictest26720-21tr10:20:152
1350599129,2cyclictest25304-21sort12:15:162
1350599129,2cyclictest23923-21cut07:30:162
1350599129,2cyclictest22050-21tr08:35:142
1350599129,2cyclictest19960-21head10:35:192
1350599129,2cyclictest19960-21head10:35:182
1350599129,2cyclictest14963-21tr08:20:122
1350599129,2cyclictest14433-21chrt11:10:212
1350599129,2cyclictest0-21swapper/209:47:232
1350599128,3cyclictest16369-21wc11:50:202
1350599126,1cyclictest0-21swapper/211:30:132
1350599122,9cyclictest4718-21/usr/sbin/munin11:05:212
13505991211,1cyclictest32-21ksoftirqd/211:04:002
1350599121,10cyclictest6739-21wget09:10:152
13505991210,1cyclictest0-21swapper/211:15:572
1350599120,8cyclictest0-21swapper/209:21:112
1350299121,10cyclictest0-21swapper/108:40:181
1349999129,2cyclictest8589-21sort08:05:200
1349999129,2cyclictest7761-21tr09:30:130
1349999129,2cyclictest6720-21wc08:00:250
1349999129,2cyclictest6708-21sed11:05:280
1349999129,2cyclictest5191-21grep09:05:230
1349999129,2cyclictest29774-21grep07:40:260
1349999129,2cyclictest19224-21grep07:20:160
1349999129,2cyclictest1607-21sed07:50:220
1349999128,3cyclictest13841-21cat11:29:590
1349999127,1cyclictest141rcu_preempt07:49:590
1349999122,1cyclictest141rcu_preempt10:14:220
1349999120,12cyclictest0-21swapper/012:28:260
1349999120,12cyclictest0-21swapper/012:19:210
1349999120,11cyclictest11475-21grep10:30:170
1350899119,1cyclictest5553-21grep11:05:173
1350899119,1cyclictest27633-21head08:45:193
1350899119,1cyclictest27633-21head08:45:183
1350899118,2cyclictest6302-21ssh10:26:563
1350899118,2cyclictest5310-21ssh12:22:393
1350899118,2cyclictest32686-21ssh09:45:113
1350899118,2cyclictest32578-21wc08:55:203
1350899118,2cyclictest3222-21expr07:55:153
1350899118,2cyclictest32019-21cut09:25:153
1350899118,2cyclictest31351-21sh08:55:003
1350899118,2cyclictest31067-21sh11:00:383
1350899118,2cyclictest29112-21cat07:40:193
1350899118,2cyclictest28231-21wc10:20:263
1350899118,2cyclictest27747-21grep10:40:153
1350899118,2cyclictest26720-21cat07:35:213
1350899118,2cyclictest25814-21ssh11:36:253
1350899118,2cyclictest25341-21ssh12:35:073
1350899118,2cyclictest20878-21cat10:35:273
1350899118,2cyclictest20878-21cat10:35:263
1350899118,2cyclictest18602-21rm11:51:333
1350899118,2cyclictest17285-21grep08:25:153
1350899118,2cyclictest1705-21cut10:05:153
1350899118,2cyclictest15020-21head11:30:163
1350899118,2cyclictest13367-21ssh12:27:253
1350899118,2cyclictest10352-21ssh09:30:593
1350899118,1cyclictest9726-21ssh10:10:143
1350899117,3cyclictest25981-21cat08:40:283
1350899111,9cyclictest0-21swapper/311:16:253
1350899110,9cyclictest0-21swapper/309:40:363
1350899110,10cyclictest0-21swapper/310:00:133
1350599118,2cyclictest8396-21awk11:45:232
1350599118,2cyclictest3477-21ssh10:05:322
1350599118,2cyclictest31688-21grep08:55:142
1350599118,2cyclictest27534-21ssh11:37:452
1350599118,2cyclictest25764-21ssh11:55:352
1350599118,2cyclictest2518-21uname09:00:212
1350599118,2cyclictest2496-21chrt07:54:522
1350599118,2cyclictest22770-21ssh10:55:432
1350599118,2cyclictest20916-21grep07:25:142
1350599118,2cyclictest19664-21ssh12:11:372
1350599118,2cyclictest14511-21wc08:20:012
1350599118,2cyclictest11981-21chrt08:14:262
1350599118,2cyclictest10928-21grep10:10:212
1350599118,2cyclictest10897-21ssh12:25:292
1350599112,2cyclictest0-21swapper/210:04:162
1350599111,8cyclictest17824-21grep12:30:172
1350599110,11cyclictest0-21swapper/207:37:332
1350599110,10cyclictest20568-21taskset07:24:332
1349999118,2cyclictest4432-21sed07:55:270
1349999118,2cyclictest31075-21ssh10:22:280
1349999118,2cyclictest30018-21head08:50:170
1349999118,2cyclictest26497-21hostname07:35:180
1349999118,2cyclictest23895-21sh10:56:370
1349999118,2cyclictest23533-21tr07:30:150
1349999118,2cyclictest22987-21sh07:30:000
1349999118,2cyclictest22911-21ssh10:36:500
1349999118,2cyclictest22911-21ssh10:36:490
1349999118,2cyclictest20718-21uname08:30:220
1349999118,2cyclictest20682-21ssh12:12:240
1349999118,2cyclictest18072-21ssh11:31:540
1349999118,2cyclictest14447-21readlink08:20:000
1349999118,2cyclictest12496-21ssh09:51:520
1349999117,3cyclictest94-21kswapd009:40:120
1349999117,3cyclictest6657-21basename09:10:150
1350899108,1cyclictest5611-21cat08:00:153
1350899108,1cyclictest4394-21expr09:05:163
1350899108,1cyclictest16884-21grep07:15:183
1350899108,1cyclictest15534-21cat08:20:163
1350899107,2cyclictest30782-21cat07:45:133
1350899107,2cyclictest18650-21grep07:20:133
1350599108,1cyclictest30673-21wc07:45:132
1350599107,2cyclictest9238-21head08:05:262
1350599100,8cyclictest17216-21cut07:15:212
1349999108,1cyclictest4933-21ntpq10:45:190
1349999108,1cyclictest27252-21ssh12:35:270
1349999108,1cyclictest236092sleep008:35:560
1349999108,1cyclictest15061-21cat08:20:140
1349999108,1cyclictest11942-21chrt08:13:560
1349999107,2cyclictest31569-21ssh09:25:070
1349999107,2cyclictest28525-21sed08:45:270
1349999107,2cyclictest28525-21sed08:45:270
1349999107,2cyclictest18219-21tr08:25:200
1349999107,2cyclictest1446-21date09:00:000
1349999100,9cyclictest36612sleep009:04:380
135059997,1cyclictest26984-21awk08:45:152
135059997,1cyclictest26984-21awk08:45:142
135059994,4cyclictest29249-21apt-get08:50:132
134999996,2cyclictest13844-21cut07:10:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional