You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-14 - 04:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Mon Sep 14, 2026 00:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2310599594,41cyclictest27187-21ssh21:20:170
23128995337,9cyclictest2917-21apt-get20:30:142
23128994934,4cyclictest11537-21apt-get23:35:132
23128994818,21cyclictest0-21swapper/219:40:152
2310599485,8cyclictest131rcu_preempt21:55:130
2310599472,39cyclictest21946-21ssh21:35:120
23128994431,9cyclictest9487-21apt-get22:40:152
23128994415,20cyclictest20175-21ssh22:10:092
2314099436,26cyclictest35966irq/58-eth022:05:153
23140994318,17cyclictest35966irq/58-eth021:10:133
23128994315,18cyclictest9069-21ssh21:10:132
23128994314,10cyclictest3982-21sh00:24:472
2311699432,31cyclictest11151-21ssh22:05:151
2314099424,28cyclictest35966irq/58-eth022:35:143
23128994234,3cyclictest4567-21apt-get00:25:132
2312899422,21cyclictest13259-21ssh23:00:142
23128994133,3cyclictest333-21apt-get22:35:142
23128994127,10cyclictest11139-21apt-get22:05:152
23128994119,17cyclictest0-21swapper/219:10:152
23128994033,4cyclictest28038-21apt-get19:15:142
23128994023,12cyclictest9862-21apt-get00:10:182
23128994017,12cyclictest0-21swapper/223:10:122
23128994013,13cyclictest21249-21irqrtprio23:40:192
23128993924,10cyclictest18921-21apt-get00:15:152
23128993918,18cyclictest18504-21/usr/sbin/munin22:45:162
2310599391,27cyclictest13277-21ssh23:00:150
23128993831,4cyclictest392-21apt-get19:20:132
23128993824,9cyclictest0-21swapper/223:25:132
23128993823,11cyclictest0-21swapper/223:20:122
23128993821,13cyclictest6872-21ssh23:50:172
23128993818,17cyclictest2522-21/usr/sbin/munin23:30:292
23128993818,15cyclictest29216-21/usr/sbin/munin22:15:252
23128993812,12cyclictest7090-21ssh21:45:002
23128993811,13cyclictest7902-21apt-get23:15:132
23128993811,11cyclictest0-21swapper/222:26:032
23128993717,12cyclictest17554-21timerandwakeup00:30:292
23128993717,12cyclictest17554-21timerandwakeup00:30:292
23128993717,11cyclictest21137-21sed20:15:182
23128993712,8cyclictest30253-21ssh21:39:422
23128993712,12cyclictest1920-21unixbench_multi20:25:282
2310599373,28cyclictest11568-21ssh23:35:140
2310599371,25cyclictest30937-21ssh21:40:130
23140993615,12cyclictest0-21swapper/323:05:143
2314099361,20cyclictest35966irq/58-eth000:05:133
23128993617,15cyclictest0-21swapper/200:05:142
23128993613,9cyclictest15130-21apt-get23:55:112
23128993612,9cyclictest0-21swapper/222:34:262
23128993612,11cyclictest11140-21ssh21:28:312
2310599361,24cyclictest20235-21ssh22:10:120
2312899359,11cyclictest0-21swapper/220:20:292
23128993524,7cyclictest25582-21apt-get21:55:132
23128993522,11cyclictest11279-21mailstats20:35:262
23128993514,12cyclictest16232-21/usr/sbin/munin20:10:162
23128993511,10cyclictest0-21swapper/222:03:522
23128993510,8cyclictest0-21swapper/221:34:332
2310599352,19cyclictest789-21ssh00:05:130
2314099348,16cyclictest35966irq/58-eth020:15:153
2312899349,10cyclictest0-21swapper/221:15:272
2312899348,11cyclictest0-21swapper/222:52:382
23128993419,12cyclictest31714-21munin-node21:00:112
23128993418,9cyclictest13780-21ls19:30:292
23128993418,13cyclictest32754-21processes19:50:272
23128993418,13cyclictest28209-21sed19:45:282
23128993416,12cyclictest27-21ksoftirqd/200:00:182
23128993412,13cyclictest11471-21fschecks_time20:05:162
23128993410,19cyclictest29058-21ssh00:37:282
23128993410,19cyclictest29058-21ssh00:37:272
2311699342,3cyclictest131rcu_preempt23:35:131
2311699342,21cyclictest25599-21ssh21:55:151
2311699341,25cyclictest4577-21ssh00:25:121
23116993412,19cyclictest21-21ksoftirqd/122:35:141
2314099335,13cyclictest35966irq/58-eth022:30:183
2314099331,19cyclictest35966irq/58-eth020:05:133
23128993319,7cyclictest6638-21forks20:00:172
23128993318,8cyclictest18732-21uptime19:35:292
23128993318,12cyclictest13801-21ssh22:24:212
23128993318,12cyclictest13801-21ssh22:24:212
23128993317,8cyclictest24241-21tr23:05:222
23128993317,13cyclictest9940-21date19:30:012
23128993317,13cyclictest26608-21cat20:50:322
23128993316,13cyclictest5258-21grep19:55:252
23128993316,13cyclictest30309-21sh23:45:192
23128993316,13cyclictest17320-21/usr/sbin/munin20:45:262
2311699336,23cyclictest6163-21ssh23:50:111
23128993217,12cyclictest31072-21kworker/u8:221:45:032
23128993216,13cyclictest30996-21awk21:20:312
23128993215,13cyclictest12839-21cron23:00:012
23128993210,10cyclictest24137-21ssh21:53:452
23116993215,14cyclictest22251-21ssh23:05:141
23116993212,12cyclictest21-21ksoftirqd/119:20:131
2310599323,22cyclictest18544-21ssh22:45:130
23105993112,10cyclictest23092-21cyclictest00:30:140
23105993112,10cyclictest23092-21cyclictest00:30:140
2314099301,18cyclictest35966irq/58-eth020:40:123
23128993015,12cyclictest12510-21/usr/sbin/munin20:40:322
2314099291,17cyclictest35966irq/58-eth022:55:123
23128992914,12cyclictest4230-21grep21:05:152
2311699297,19cyclictest14764-21ssh22:25:131
23128992816,8cyclictest30371-21ntpq20:55:252
2311699287,16cyclictest12803-21ssh21:30:131
2311699282,22cyclictest15159-21ssh23:55:121
23116992815,9cyclictest21-21ksoftirqd/121:05:121
2310599282,19cyclictest376666ssh21:25:120
2310599281,20cyclictest18112-21ssh21:15:150
2314099271,16cyclictest0-21swapper/321:45:133
2311699272,21cyclictest29248-21rm22:15:131
23116992713,9cyclictest0-21swapper/100:15:151
2310599274,16cyclictest23876-21sh22:30:180
2310599273,18cyclictest20574-21ssh23:40:130
2310599272,19cyclictest20731-21rm23:20:260
23105992719,5cyclictest8671-21diskmemload23:15:130
2310599271,20cyclictest11169-21rm22:05:160
2310599270,19cyclictest22249-21sh23:05:140
2314099263,14cyclictest35966irq/58-eth000:10:173
2311699261,20cyclictest31582-21ssh00:20:241
2310599261,20cyclictest6179-21ssh23:50:110
2311699255,12cyclictest9874-21ssh00:10:171
2311699252,19cyclictest17001-21ssh23:20:141
23116992518,5cyclictest211ksoftirqd/121:50:141
2310599252,18cyclictest4195-21ssh22:55:130
2310599251,20cyclictest9518-21ssh22:40:160
2314099241,13cyclictest35966irq/58-eth021:25:123
2314099240,14cyclictest35966irq/58-eth023:50:123
23128992413,8cyclictest23170-21sort19:10:012
2311699249,3cyclictest131rcu_preempt23:30:151
2311699249,11cyclictest21-21ksoftirqd/123:48:061
2311699241,17cyclictest5834-21ssh22:20:131
2311699241,17cyclictest5834-21ssh22:20:131
2310599242,14cyclictest27545-21ssh22:50:110
2310599241,21cyclictest8671-21diskmemload00:00:160
2310599241,18cyclictest18933-21rm00:15:140
23105992411,7cyclictest7584-21ssh21:45:130
23140992312,7cyclictest35966irq/58-eth023:10:123
2311699233,15cyclictest21-21ksoftirqd/119:30:131
2311699232,6cyclictest0-21swapper/123:28:271
2311699231,19cyclictest26294-21sshd00:35:271
2311699231,19cyclictest26294-21sshd00:35:271
2310599232,19cyclictest8671-21diskmemload00:24:270
23105992317,3cyclictest0-21swapper/022:35:150
2310599231,19cyclictest13323-21ls00:30:020
2314099223,12cyclictest0-21swapper/300:25:143
23140992210,9cyclictest0-21swapper/322:30:003
2311699227,12cyclictest791-21apt-get00:05:131
2311699227,12cyclictest7874-21grep21:25:291
2311699222,17cyclictest30932-21apt-get21:40:121
2311699222,16cyclictest24163-21needreboot23:40:241
2311699222,15cyclictest7913-21ssh22:55:271
2311699221,17cyclictest9517-21sh22:40:151
2311699221,14cyclictest167-21systemd-network21:15:151
2311699220,18cyclictest6520-21find20:00:141
2310599222,17cyclictest2562-21ssh23:30:140
2310599221,16cyclictest16607-21ssh21:50:150
2310599221,14cyclictest17888-21ls21:15:020
2310599220,18cyclictest19614-21ls19:40:020
2314099219,9cyclictest0-21swapper/322:45:543
2314099218,8cyclictest35966irq/58-eth021:55:153
2314099213,12cyclictest0-21swapper/323:20:273
2311699218,3cyclictest131rcu_preempt22:45:251
2311699213,12cyclictest23892-21ssh22:30:191
23116992115,4cyclictest0-21swapper/123:10:111
2311699211,16cyclictest7878-21ls21:45:171
2311699211,15cyclictest23340-21ls19:40:291
2310599212,16cyclictest9880-21ssh00:10:160
2310599211,15cyclictest31914-21cut21:00:160
2314099204,11cyclictest35966irq/58-eth020:00:143
2314099202,13cyclictest0-21swapper/323:35:143
23140992013,5cyclictest35966irq/58-eth023:45:143
23140992011,7cyclictest0-21swapper/300:31:063
23140992011,7cyclictest0-21swapper/300:31:063
2311699202,15cyclictest27534-21apt-get22:50:091
23116992012,6cyclictest30810-21sshd21:20:271
23116992012,6cyclictest27968-21sh00:00:311
2311699201,16cyclictest8635-21ssh23:15:181
2311699201,15cyclictest9065-21apt-get21:10:131
2311699201,10cyclictest2093-21/usr/sbin/munin22:00:261
23116992010,7cyclictest20754-21sed22:10:171
2310599203,8cyclictest0-21swapper/019:45:140
23105992014,4cyclictest0-21swapper/019:25:140
23105992013,5cyclictest0-21swapper/023:10:150
2310599201,14cyclictest4783-21kthreadcore20:30:220
2310599201,13cyclictest31239-21apt-get00:40:010
2310599201,13cyclictest31239-21apt-get00:40:010
2310599201,13cyclictest29059-21ls20:20:220
2314099193,11cyclictest0-21swapper/300:23:443
2314099192,11cyclictest0-21swapper/320:55:143
2314099191,13cyclictest0-21swapper/319:15:023
2314099191,12cyclictest0-21swapper/319:20:273
23116991914,3cyclictest28145-21grep19:15:161
23116991914,3cyclictest0-21swapper/121:35:051
23116991913,4cyclictest0-21swapper/119:25:301
23116991911,5cyclictest9485-21cut20:35:181
2311699191,15cyclictest22229-21cut20:50:171
23116991911,5cyclictest14959-21kthreadcore00:30:211
23116991911,5cyclictest14959-21kthreadcore00:30:201
2310599198,3cyclictest131rcu_preempt22:20:160
2310599198,3cyclictest131rcu_preempt22:20:160
2310599193,2cyclictest0-21swapper/023:55:110
2310599193,11cyclictest0-21swapper/019:40:150
2310599192,9cyclictest0-21swapper/022:27:310
2310599192,13cyclictest30270-21sed23:45:180
2310599191,9cyclictest32676-21/usr/sbin/munin22:15:230
2310599191,16cyclictest5166-21ls19:25:020
23105991911,6cyclictest1787-21sshd20:25:270
23105991911,6cyclictest1-21systemd20:47:160
2310599190,15cyclictest4094-21kthreadcore22:00:210
2310599190,12cyclictest16803-21grep20:10:200
2314099189,7cyclictest0-21swapper/321:54:333
2314099189,6cyclictest0-21swapper/322:24:303
2314099189,6cyclictest0-21swapper/322:24:303
2314099189,6cyclictest0-21swapper/322:16:083
2314099189,6cyclictest0-21swapper/322:13:163
2314099189,6cyclictest0-21swapper/322:00:463
2314099189,6cyclictest0-21swapper/321:40:273
2314099189,4cyclictest0-21swapper/300:01:143
2314099188,8cyclictest0-21swapper/321:37:583
2314099187,8cyclictest0-21swapper/323:00:413
2314099183,10cyclictest0-21swapper/323:27:533
2314099183,10cyclictest0-21swapper/322:40:273
2314099182,10cyclictest0-21swapper/319:55:043
2314099182,10cyclictest0-21swapper/300:15:053
23140991812,4cyclictest35966irq/58-eth023:30:223
2314099181,13cyclictest0-21swapper/320:45:183
2314099181,11cyclictest0-21swapper/319:45:283
23140991810,6cyclictest0-21swapper/323:15:453
23140991810,6cyclictest0-21swapper/321:23:513
2311699189,7cyclictest30323-21sh20:55:231
2311699182,12cyclictest32648-21ntp_192.168.11419:50:251
2311699181,14cyclictest5833-21kthreadcore20:30:231
2311699181,14cyclictest31732-21apt-get21:00:121
2311699181,14cyclictest27225-21kthreadcore19:45:221
2311699181,14cyclictest17111-21munin-run20:45:011
2311699181,13cyclictest17096-21awk23:00:291
2310599189,7cyclictest32603-21cron19:20:010
2310599181,9cyclictest16135-21cut21:30:230
2314099179,6cyclictest0-21swapper/323:56:353
2314099179,6cyclictest0-21swapper/323:44:513
2314099177,7cyclictest35966irq/58-eth021:15:253
2314099172,10cyclictest0-21swapper/319:40:153
2314099171,10cyclictest0-21swapper/300:35:163
2314099171,10cyclictest0-21swapper/300:35:163
2311699176,9cyclictest17826-21kthreadcore19:35:221
2311699175,3cyclictest131rcu_preempt20:10:011
2311699174,10cyclictest22426-21kthreadcore20:15:211
2311699174,10cyclictest18523-21kthreadcore20:45:201
2311699174,10cyclictest1735-21switchtime20:25:261
2311699172,11cyclictest1886-21sort19:55:141
2311699171,11cyclictest26670-21sed19:10:261
2310599178,6cyclictest11397-21timerandwakeup20:35:270
2310599177,8cyclictest15953-21date20:10:010
2310599172,12cyclictest14740-21latency_hist19:35:010
2310599172,10cyclictest0-21swapper/019:10:160
2310599171,9cyclictest21136-21grep20:15:180
2310599171,9cyclictest12636-21cut20:40:150
2314099168,6cyclictest0-21swapper/321:30:483
2314099162,9cyclictest35966irq/58-eth021:05:123
2314099161,12cyclictest0-21swapper/320:35:273
23140991610,4cyclictest0-21swapper/322:51:013
2311699162,11cyclictest16329-21grep20:10:171
2311699161,12cyclictest26324-21irqrtprio20:20:181
2310599166,4cyclictest23092-21cyclictest19:55:110
2310599165,4cyclictest26277-21hwlatdetect23:25:170
2310599161,13cyclictest4122-21apt-get21:05:120
23105991611,3cyclictest27201-21if_err_eth120:55:170
2310599161,11cyclictest22214-21diskstats20:50:170
2310599160,13cyclictest29260-21apt-get19:50:120
2310599160,13cyclictest0-21swapper/020:00:160
23140991510,3cyclictest0-21swapper/319:28:273
2311699141,10cyclictest23147-21latency_hist19:10:011
2310599141,10cyclictest23148-21latency_hist19:10:010
2314099138,3cyclictest0-21swapper/320:25:283
2314099138,3cyclictest0-21swapper/320:20:183
2314099131,9cyclictest0-21swapper/319:50:133
2314099131,7cyclictest0-21swapper/320:50:293
2314099131,5cyclictest0-21swapper/319:30:193
2314099121,9cyclictest0-21swapper/321:00:183
2314099121,9cyclictest0-21swapper/320:30:223
2314099121,9cyclictest0-21swapper/319:35:153
2314099121,9cyclictest0-21swapper/319:10:013
2314099112,5cyclictest0-21swapper/319:15:143
2314099111,6cyclictest0-21swapper/320:10:173
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional