You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-19 - 13:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Thu Mar 19, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
539329772,6sleep00-21swapper/019:07:100
537029060,10sleep20-21swapper/219:06:522
554027958,6sleep10-21swapper/119:09:041
540227966,9sleep30-21swapper/319:07:173
188022740,1sleep125-21ksoftirqd/121:21:181
152682610,0sleep30-21swapper/319:30:013
570899325,2cyclictest21445-21cut23:00:142
570899305,20cyclictest30632-21kworker/u8:121:51:512
5712992827,1cyclictest0-21swapper/321:26:523
570899260,1cyclictest0-21swapper/221:10:142
570899253,9cyclictest14000-21apt-get23:15:112
570899250,24cyclictest0-21swapper/222:44:552
570899240,3cyclictest0-21swapper/223:29:272
571299237,15cyclictest4900-31find00:07:473
570899230,2cyclictest0-21swapper/223:59:312
570899230,0cyclictest0-21swapper/223:52:302
570899222,2cyclictest27925-21users22:05:262
570899220,7cyclictest0-21swapper/221:25:172
570899220,21cyclictest0-21swapper/221:21:102
570899220,19cyclictest0-21swapper/200:17:212
5705992221,1cyclictest0-21swapper/120:30:101
570899210,4cyclictest2332-21runrttasks20:10:112
570899210,3cyclictest16550-21tail23:35:212
570899210,3cyclictest0-21swapper/200:30:132
570899210,3cyclictest0-21swapper/200:30:122
570899210,2cyclictest0-21swapper/221:48:062
570599212,3cyclictest141rcu_preempt21:51:041
5705992120,1cyclictest0-21swapper/121:16:441
5712992018,2cyclictest0-21swapper/323:22:433
570899201,4cyclictest30853-21diskmemload21:42:422
570899200,4cyclictest0-21swapper/222:35:142
570899200,18cyclictest0-21swapper/200:20:132
570599203,11cyclictest0-21swapper/120:45:121
570599200,1cyclictest0-21swapper/123:08:321
570899192,3cyclictest0-21swapper/221:30:142
570899191,5cyclictest11021-21sendmail_mailqu20:25:222
570899191,1cyclictest30853-21diskmemload22:57:132
570899191,0cyclictest0-21swapper/222:10:082
570899190,2cyclictest0-21swapper/222:26:052
570899190,0cyclictest0-21swapper/223:08:362
5705991918,1cyclictest30853-21diskmemload22:43:291
570399192,16cyclictest0-21swapper/022:35:360
5703991917,2cyclictest13-21ksoftirqd/021:44:420
570899185,12cyclictest258692sleep200:39:192
570899181,2cyclictest0-21swapper/219:35:122
570899181,16cyclictest0-21swapper/222:15:102
570899180,3cyclictest0-21swapper/223:10:132
570899180,15cyclictest0-21swapper/222:21:172
5705991818,0cyclictest0-21swapper/123:40:131
570599180,18cyclictest0-21swapper/123:25:331
570599180,18cyclictest0-21swapper/100:02:231
570399181,7cyclictest225-21systemd-journal20:40:000
570899175,1cyclictest30853-21diskmemload00:26:282
570899171,6cyclictest225-21systemd-journal00:06:572
5705991716,1cyclictest0-21swapper/121:25:031
570599170,17cyclictest0-21swapper/123:55:231
111302170,0sleep20-21swapper/221:36:012
5712991613,2cyclictest21683-21sh23:19:533
571299160,1cyclictest0-21swapper/321:22:563
570899169,6cyclictest15179-21cron19:30:002
570899168,7cyclictest6414-21rm23:49:092
570899168,7cyclictest17850-21tr22:00:132
570899168,7cyclictest14991-21df_abs20:35:162
570899167,8cyclictest31843-21idleruntime-cro20:05:002
5708991614,1cyclictest0-21swapper/223:22:032
570899161,2cyclictest30853-21diskmemload22:45:092
570899160,16cyclictest0-21swapper/221:15:152
5703991615,1cyclictest0-21swapper/021:57:570
570399161,2cyclictest0-21swapper/023:59:220
570899159,5cyclictest6820-21cron20:20:002
570899158,6cyclictest6523-21sh22:50:262
570899158,6cyclictest6523-21sh22:50:262
570899158,6cyclictest30367-21ssh23:44:052
570899158,6cyclictest20860-21cut19:40:162
570899158,6cyclictest15724-21grep19:30:142
570899157,7cyclictest27065-21cron19:55:002
570899155,9cyclictest0-21swapper/220:55:262
5708991514,1cyclictest0-21swapper/222:33:162
570899151,13cyclictest0-21swapper/221:00:202
570899150,14cyclictest1233-21sort20:05:252
570899150,13cyclictest0-21swapper/221:55:492
5705991514,1cyclictest0-21swapper/121:35:341
5705991511,3cyclictest1884-21wc00:05:171
5705991510,3cyclictest4004-21ssh22:50:001
570399155,7cyclictest18944-21apt-get00:35:130
570399155,5cyclictest2497-21awk22:29:590
5703991513,2cyclictest13-21ksoftirqd/022:55:280
571299146,8cyclictest3084-21kworker/3:119:20:333
5712991411,2cyclictest31525-21basename23:45:003
5712991411,2cyclictest11112-21sed22:15:153
571299140,12cyclictest8582-21tr19:15:143
571299140,12cyclictest8582-21tr19:15:143
570899148,5cyclictest15099-21ssh00:13:202
570899147,2cyclictest8142-21wc20:20:182
570899147,2cyclictest25805-21chrt00:00:192
570899147,2cyclictest22624-21cut19:45:132
570899147,2cyclictest19569-21expr20:45:142
570899147,2cyclictest18501-21grep20:40:222
570899142,2cyclictest225-21systemd-journal19:15:002
570899141,2cyclictest225-21systemd-journal19:20:002
570899141,2cyclictest225-21systemd-journal19:20:002
5708991410,3cyclictest8214-21tr23:30:202
570899140,6cyclictest225-21systemd-journal20:53:362
570599148,5cyclictest20572-21tail22:20:271
5705991413,1cyclictest0-21swapper/121:45:201
5705991411,2cyclictest19728-21tail00:35:171
5705991410,3cyclictest25-21ksoftirqd/123:24:411
570599140,13cyclictest14820-21grep22:55:261
570599140,13cyclictest0-21swapper/121:32:341
570399140,1cyclictest0-21swapper/023:29:230
571299139,3cyclictest39-21ksoftirqd/320:20:113
571299137,5cyclictest8615-21/usr/sbin/munin21:15:193
571299135,3cyclictest225-21systemd-journal23:30:263
5712991311,2cyclictest28257-21kworker/3:222:29:533
571299131,10cyclictest0-21swapper/321:51:413
5712991310,2cyclictest4369-21wc22:30:183
5712991310,2cyclictest30397-21grep22:45:263
5712991310,2cyclictest23242-21grep20:50:263
5712991310,2cyclictest23142-21df19:45:153
5712991310,2cyclictest10607-21grep21:55:203
5712991310,2cyclictest10093-21timerandwakeup21:35:243
571299130,1cyclictest0-21swapper/321:44:503
571299130,12cyclictest56550irq/124-enp0s3100:14:273
571299130,12cyclictest27583-21grep00:20:213
571299130,12cyclictest12491-21sed00:30:253
571299130,12cyclictest12491-21sed00:30:253
570899139,3cyclictest30435-21grep21:05:262
570899136,1cyclictest12774-21tr20:30:142
570899130,3cyclictest225-21systemd-journal19:56:112
570599139,3cyclictest26990-21cut00:20:161
570599139,3cyclictest17804-21cut00:15:121
570599131,1cyclictest0-21swapper/100:32:301
570599131,1cyclictest0-21swapper/100:32:301
5705991311,1cyclictest0-21swapper/120:05:141
5705991310,2cyclictest18753-21wc22:00:171
5705991310,2cyclictest1511-21grep23:45:251
5705991310,2cyclictest13148-21tr22:35:191
570599130,3cyclictest0-21swapper/100:25:121
570599130,1cyclictest0-21swapper/122:51:461
570599130,1cyclictest0-21swapper/122:51:461
570599130,1cyclictest0-21swapper/122:27:261
570399139,3cyclictest24809-21cat19:50:010
5703991310,3cyclictest1978-21kworker/u8:023:09:480
5703991310,2cyclictest5065-21ssh00:07:060
5703991310,2cyclictest29809-21ntpq22:45:210
5703991310,2cyclictest21368-21cut23:00:140
5703991310,2cyclictest12354-21grep19:20:250
5703991310,2cyclictest11385-21grep00:30:160
5703991310,2cyclictest11385-21grep00:30:160
571299129,2cyclictest8670-21uniq23:50:153
571299129,2cyclictest3141-21ssh00:25:173
571299129,2cyclictest30483-21tail23:05:173
571299129,2cyclictest28257-21kworker/3:221:12:333
571299129,2cyclictest19931-21awk22:20:213
571299129,2cyclictest18994-21tr00:35:133
571299129,2cyclictest14852-21grep22:55:263
571299129,2cyclictest1398-21wc21:30:213
571299129,2cyclictest12541-21head22:35:163
571299128,2cyclictest22767-21sh23:00:223
571299123,5cyclictest20227-21apt-get19:40:123
571299122,9cyclictest0-21swapper/300:00:523
5712991211,1cyclictest0-21swapper/321:00:133
5712991210,1cyclictest18493-21grep20:40:233
571299120,12cyclictest0-21swapper/323:46:553
570899121,2cyclictest225-21systemd-journal19:25:002
570599129,2cyclictest9703-21chrt20:25:121
570599129,2cyclictest9496-21grep21:55:141
570599129,2cyclictest8535-21tr19:15:141
570599129,2cyclictest8535-21tr19:15:141
570599129,2cyclictest8041-21idleruntime-cro19:15:001
570599129,2cyclictest7582-21tail23:10:261
570599129,2cyclictest3432-21cat20:10:191
570599129,2cyclictest28066-21grep19:55:161
570599129,2cyclictest23311-21tail19:45:171
570599129,2cyclictest23276-21grep20:50:271
570599129,2cyclictest21920-21head19:40:261
570599129,2cyclictest1938-21cut22:10:131
570599129,2cyclictest19052-21grep19:35:201
570599129,2cyclictest10054-21grep00:10:151
570599128,2cyclictest2068-21sh21:11:161
570599128,2cyclictest15546-21grep23:15:211
570599126,5cyclictest4233-21tr22:30:181
570599126,5cyclictest25460-21grep19:50:151
570599121,3cyclictest27420-21uname22:05:221
570399129,2cyclictest900-21grep21:10:270
570399129,2cyclictest6915-21cat19:10:170
570399129,2cyclictest6821-21cron20:20:000
570399129,2cyclictest3344-21grep22:30:130
570399129,2cyclictest30504-21grep20:00:180
570399129,2cyclictest30224-21apt-get23:25:000
570399129,2cyclictest27964-21grep19:55:160
570399129,2cyclictest26980-21uniq21:00:160
570399129,2cyclictest26791-21ssh22:05:160
570399129,2cyclictest23951-21grep23:40:150
570399129,2cyclictest23666-21wc21:25:010
570399129,2cyclictest19415-21sed22:00:250
570399129,2cyclictest17946-21cut20:40:160
570399129,2cyclictest1639-21sed21:50:160
570399129,2cyclictest14778-21head23:15:160
570399129,2cyclictest10386-21chrt23:50:430
570399123,7cyclictest0-21swapper/021:15:120
5703991210,2cyclictest13-21ksoftirqd/021:33:110
570399120,6cyclictest17284-21idleruntime-cro00:15:000
571299119,2cyclictest28257-21kworker/3:223:55:333
571299119,1cyclictest16122-21cut19:30:153
571299118,2cyclictest5049-21expr20:15:133
571299118,2cyclictest4730-21sort22:50:133
571299118,2cyclictest4730-21sort22:50:133
571299118,2cyclictest31684-21sort23:25:173
571299118,2cyclictest30581-21ssh22:07:303
571299118,2cyclictest21045-21ssh00:16:263
571299118,2cyclictest17337-21rm22:00:023
571299118,2cyclictest15581-21tr23:35:163
571299118,2cyclictest10488-21ssh23:12:393
571299117,3cyclictest2591-21tr20:10:143
571299110,10cyclictest20881-21uniq22:40:153
570599118,2cyclictest28653-21wc21:05:001
570599118,2cyclictest18120-21grep21:40:211
570599118,2cyclictest16716-21tr23:35:231
570599118,2cyclictest16712-21sh22:18:541
570599118,2cyclictest15444-21sed20:35:191
570599118,2cyclictest13803-21tail19:25:161
570599118,2cyclictest12337-21ssh23:52:151
570599118,2cyclictest11378-21cut19:20:151
570599117,3cyclictest7873-21tail20:20:161
570599117,3cyclictest24749-21head20:55:171
570599117,2cyclictest6997-21cat20:20:001
570599116,5cyclictest38150irq/131-snd_hda23:31:411
570599110,10cyclictest21686-21expr23:00:151
570399118,2cyclictest9906-21sendmail_mailtr21:35:210
570399118,2cyclictest9850-21cat19:15:250
570399118,2cyclictest9850-21cat19:15:240
570399118,2cyclictest8559-21ssh23:11:080
570399118,2cyclictest6848-21ssh22:50:410
570399118,2cyclictest6848-21ssh22:50:410
570399118,2cyclictest3754-21cat20:10:250
570399118,2cyclictest3581-21ssh22:10:240
570399118,2cyclictest32451-21sh00:23:460
570399118,2cyclictest2729-21df00:25:140
570399118,2cyclictest26478-21cat19:50:250
570399118,2cyclictest24804-21grep21:25:160
570399118,2cyclictest24775-21cat00:00:120
570399118,2cyclictest22660-21ssh22:22:000
570399118,2cyclictest18598-21sed19:35:160
570399118,2cyclictest17585-21ssh00:15:050
570399118,2cyclictest1592-21chrt20:07:280
570399118,2cyclictest10475-21head20:25:160
570399117,3cyclictest26110-21chrt22:43:310
570399117,2cyclictest25832-21ssh21:45:170
570399112,6cyclictest15334-21diskmemload23:35:150
570399110,0cyclictest0-21swapper/021:05:140
571299109,1cyclictest0-21swapper/320:55:143
571299108,1cyclictest7565-21grep19:10:243
571299108,1cyclictest30957-21grep20:00:233
571299108,1cyclictest29035-21cat21:05:153
571299108,1cyclictest28245-21cat19:55:183
571299108,1cyclictest16782-21cat20:40:003
571299108,1cyclictest12309-21grep20:30:123
571299108,1cyclictest11506-21chrt20:26:143
571299107,2cyclictest4608-21ssh22:10:573
571299107,2cyclictest26743-21cat21:45:263
571299107,2cyclictest19545-21expr20:45:143
571299107,2cyclictest18097-21expr19:35:133
570599108,1cyclictest18455-21grep20:40:221
570599108,1cyclictest174012sleep119:33:031
570599107,2cyclictest31132-21python20:00:261
570399108,1cyclictest6735-21ssh23:30:030
570399108,1cyclictest21203-21chrt20:48:350
570399108,1cyclictest12215-21egrep20:30:110
570399107,2cyclictest31832-21ssh23:45:050
570399107,2cyclictest24234-21tr20:55:150
570399107,2cyclictest20228-21taskset19:40:120
570399107,2cyclictest17211-21taskset19:30:340
570399107,2cyclictest13862-21head19:25:170
570399106,3cyclictest10282-21ssh22:15:120
57129997,1cyclictest32469-21expr20:05:133
57129996,2cyclictest25197-21grep19:50:143
57059997,1cyclictest28828-21cut21:05:131
57039997,1cyclictest23400-21seq20:51:190
57039996,2cyclictest7146-21cut20:20:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional