You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-13 - 20:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Wed May 13, 2026 12:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
16039997239,23cyclictest35966irq/58-eth009:10:133
1599799633,44cyclictest14864-21ssh12:35:120
1599799624,48cyclictest5868-21ssh12:30:130
1601299545,7cyclictest131rcu_preempt08:40:121
16012995423,5cyclictest21-21ksoftirqd/111:05:111
1601299525,8cyclictest131rcu_preempt11:20:131
1601299524,5cyclictest131rcu_preempt12:25:121
1601299522,7cyclictest131rcu_preempt11:00:121
16039995112,24cyclictest35966irq/58-eth011:05:123
1602599491,8cyclictest27-21ksoftirqd/209:50:112
16025994915,15cyclictest27-21ksoftirqd/209:45:132
1601299481,38cyclictest31199-21ssh10:20:121
1602599471,36cyclictest29408-21ssh10:55:132
1602599462,20cyclictest20834-21kworker/u8:211:10:152
1601299463,34cyclictest7850-21ssh10:25:131
1601299462,32cyclictest11402-21ssh12:15:131
1601299451,33cyclictest9347-21ssh09:50:121
1599799451,29cyclictest4150-21ssh10:05:130
1603999432,26cyclictest35966irq/58-eth008:40:143
1601299432,5cyclictest131rcu_preempt12:05:121
16012994216,19cyclictest21-21ksoftirqd/111:30:121
16012994125,11cyclictest21-21ksoftirqd/109:35:141
1601299401,27cyclictest25834-21ssh10:35:121
1599799402,27cyclictest23803-21sh09:40:130
15997994019,16cyclictest12-21ksoftirqd/008:45:130
1603999394,24cyclictest35966irq/58-eth011:15:123
1602599382,28cyclictest11411-21ssh12:15:122
1602599381,28cyclictest18400-21ssh09:55:132
1602599381,25cyclictest4148-21sh10:05:132
1602599362,18cyclictest4410-21ssh12:28:442
16012993616,17cyclictest21-21ksoftirqd/110:50:131
16012993616,17cyclictest21-21ksoftirqd/110:50:121
1601299353,19cyclictest21-21ksoftirqd/109:10:121
1601299342,4cyclictest131rcu_preempt11:55:111
1603999324,15cyclictest35966irq/58-eth010:55:133
1603999323,17cyclictest35966irq/58-eth008:00:133
16039993219,8cyclictest35966irq/58-eth011:25:123
16025993215,11cyclictest1496-21diskmemload10:40:132
1599799324,24cyclictest22084-21sh10:15:110
1599799322,19cyclictest664-21ntpd08:55:130
1603999315,13cyclictest35966irq/58-eth009:35:153
1602599313,21cyclictest0-21swapper/207:15:142
1601299311,24cyclictest5878-21ssh12:30:121
1602599302,21cyclictest0-21swapper/209:00:132
16025992919,6cyclictest0-21swapper/212:05:132
16025992912,8cyclictest537-21apt-get11:15:122
16025992910,15cyclictest0-21swapper/212:35:112
1601299291,16cyclictest22206-21ssh11:45:121
1599799291,25cyclictest11481-21rm10:45:130
1602599282,20cyclictest0-21swapper/211:25:132
1602599282,12cyclictest930-21snmpd10:25:112
1602599280,21cyclictest9589-21ssh11:20:122
1601299282,19cyclictest4172-21ssh11:35:151
1601299282,19cyclictest4172-21ssh11:35:141
1599799283,9cyclictest12-21ksoftirqd/009:25:010
1603999276,11cyclictest0-21swapper/307:40:133
1603999271,16cyclictest35966irq/58-eth011:10:153
1602599272,21cyclictest0-21swapper/208:45:132
1602599272,11cyclictest27549-21ssh11:30:122
1602599272,10cyclictest0-21swapper/207:40:142
1601299277,13cyclictest21-21ksoftirqd/108:10:121
1601299277,11cyclictest16874-21ssh10:30:131
1601299274,11cyclictest0-21swapper/112:10:141
1601299273,15cyclictest29404-21sshd10:55:121
1601299272,19cyclictest13164-21ssh11:40:131
16012992717,6cyclictest0-21swapper/112:20:131
15997992713,10cyclictest20921-21apt-get07:15:120
1599799271,22cyclictest29075-21rm09:25:120
1603999262,14cyclictest35966irq/58-eth009:40:123
1602599264,4cyclictest131rcu_preempt08:30:132
1602599264,4cyclictest131rcu_preempt08:30:132
1602599263,16cyclictest27346irq/40-dwc2_hso07:45:112
1602599261,16cyclictest0-21swapper/212:30:122
16012992610,12cyclictest2493-21diskmemload10:40:121
15997992612,10cyclictest31187-21apt-get10:20:140
1603999254,15cyclictest36066irq/59-eth010:40:113
1603999253,17cyclictest35966irq/58-eth008:30:143
1603999253,17cyclictest35966irq/58-eth008:30:133
1603999253,15cyclictest35966irq/58-eth010:30:133
1603999252,12cyclictest35966irq/58-eth009:50:103
1602599258,7cyclictest27-21ksoftirqd/211:52:102
1602599256,11cyclictest27-21ksoftirqd/208:00:242
1602599253,13cyclictest27346irq/40-dwc2_hso10:35:112
1602599251,13cyclictest0-21swapper/210:10:132
1601299254,13cyclictest630-21Async09:36:311
1601299253,14cyclictest23787-21ssh09:40:121
1599799257,8cyclictest12-21ksoftirqd/010:38:490
1599799252,6cyclictest12-21ksoftirqd/011:18:420
15997992515,4cyclictest24009-21apt-get11:10:140
1603999242,17cyclictest35966irq/58-eth011:30:123
1602599246,4cyclictest131rcu_preempt09:23:202
1602599243,15cyclictest27346irq/40-dwc2_hso08:40:122
1602599243,14cyclictest27346irq/40-dwc2_hso08:25:122
1602599242,17cyclictest27346irq/40-dwc2_hso10:20:112
1602599242,17cyclictest1888-21ssh09:10:132
1602599242,15cyclictest27346irq/40-dwc2_hso07:30:142
1602599241,9cyclictest1586866ssh11:05:182
16025992414,8cyclictest0-21swapper/209:18:442
1602599241,12cyclictest6003-21ssh11:00:122
1599799245,3cyclictest131rcu_preempt12:05:260
1599799244,4cyclictest12-21ksoftirqd/010:55:160
1599799243,5cyclictest12-21ksoftirqd/010:28:250
1599799243,4cyclictest12-21ksoftirqd/011:25:180
1599799243,4cyclictest12-21ksoftirqd/010:40:190
1599799242,5cyclictest12-21ksoftirqd/010:50:200
1599799242,5cyclictest12-21ksoftirqd/010:50:200
1599799242,4cyclictest131rcu_preempt09:11:440
15997992418,4cyclictest151rcuc/012:22:540
15997992418,4cyclictest151rcuc/012:04:410
15997992418,2cyclictest5371-21apt-get08:40:130
15997992417,2cyclictest29305-21apt-get12:25:130
1599799241,6cyclictest12-21ksoftirqd/009:50:220
1599799241,16cyclictest22226-21ssh11:45:130
15997992410,4cyclictest12-21ksoftirqd/011:04:440
1603999233,13cyclictest0-21swapper/310:00:153
1603999232,16cyclictest35966irq/58-eth008:55:113
1602599236,6cyclictest131rcu_preempt07:25:132
1602599234,5cyclictest0-21swapper/207:35:132
1602599231,19cyclictest20748-21fschecks_time10:50:162
1602599231,19cyclictest20748-21fschecks_time10:50:152
1602599231,15cyclictest27-21ksoftirqd/211:35:152
1602599231,15cyclictest27-21ksoftirqd/211:35:142
16025992311,10cyclictest0-21swapper/209:33:472
1601299232,6cyclictest53646usb-storage07:40:051
1601299232,16cyclictest567-21ssh11:15:131
1601299231,4cyclictest27423-21ssh10:00:141
1599799236,8cyclictest12-21ksoftirqd/012:15:030
1599799234,4cyclictest12-21ksoftirqd/011:55:210
1599799233,7cyclictest151rcuc/011:50:240
1599799233,7cyclictest12-21ksoftirqd/009:37:500
1599799233,4cyclictest12-21ksoftirqd/011:35:300
1599799233,4cyclictest12-21ksoftirqd/011:35:300
1599799233,4cyclictest12-21ksoftirqd/009:15:320
1599799232,5cyclictest12-21ksoftirqd/011:20:500
15997992317,4cyclictest151rcuc/011:44:190
15997992317,4cyclictest151rcuc/010:33:330
15997992316,4cyclictest151rcuc/009:46:040
15997992316,4cyclictest151rcuc/009:30:240
1603999223,15cyclictest0-21swapper/309:55:123
16039992211,4cyclictest0-21swapper/310:20:173
16039992210,9cyclictest0-21swapper/309:20:163
1602599228,5cyclictest27-21ksoftirqd/211:48:032
1602599222,5cyclictest0-21swapper/208:55:132
1602599222,14cyclictest27346irq/40-dwc2_hso08:35:102
1602599221,5cyclictest16865-21sh10:30:122
16025992213,7cyclictest0-21swapper/207:20:182
16025992213,6cyclictest27-21ksoftirqd/212:01:052
1602599221,17cyclictest21333-21kthreadcore08:20:222
16025992211,5cyclictest12870-21ssh11:55:432
1601299224,11cyclictest0-21swapper/108:00:141
16012992211,8cyclictest17419-21apt-get07:45:111
1601299220,17cyclictest3876-21ls09:45:231
1599799226,8cyclictest12-21ksoftirqd/012:12:170
1599799224,4cyclictest12-21ksoftirqd/011:30:150
1599799224,3cyclictest131rcu_preempt10:00:130
1599799222,5cyclictest12-21ksoftirqd/010:11:220
15997992217,3cyclictest151rcuc/007:55:110
15997992216,4cyclictest151rcuc/009:57:360
15997992216,4cyclictest151rcuc/008:00:230
15997992216,3cyclictest0-21swapper/009:00:130
15997992212,7cyclictest19502-21cron08:55:020
1603999213,13cyclictest35966irq/58-eth010:45:123
1603999212,12cyclictest0-21swapper/309:45:133
16039992113,5cyclictest0-21swapper/312:30:183
16039992112,5cyclictest0-21swapper/310:50:113
16039992112,5cyclictest0-21swapper/310:50:113
1602599211,9cyclictest3648-21kthreadcore12:10:202
16025992112,6cyclictest24078-21ssh10:15:202
1602599211,18cyclictest17427-21timerandwakeup08:15:282
1602599211,11cyclictest24066-21idleruntime09:40:162
1602599211,11cyclictest21973-21latency_hist11:45:012
1602599211,10cyclictest733-21grep09:05:252
1602599211,10cyclictest30799-21grep07:55:282
16025992110,2cyclictest27-21ksoftirqd/209:35:272
1602599210,18cyclictest20422-21find12:20:122
1602599210,13cyclictest11764-21ssh10:45:172
1601299212,14cyclictest18599-21ssh11:25:131
1601299211,3cyclictest0-21swapper/109:25:291
16012992111,8cyclictest14963-21df_inode12:35:141
1601299211,17cyclictest18551-21cut09:55:151
1601299211,14cyclictest7561-21ssh11:55:021
16012992110,8cyclictest13115-21apt-get10:10:121
1599799213,6cyclictest12-21ksoftirqd/011:10:020
1599799212,12cyclictest844-21munin-node07:20:150
1599799210,16cyclictest12313-21ls08:10:230
1603999207,4cyclictest35966irq/58-eth011:45:123
1603999203,12cyclictest0-21swapper/309:15:063
1603999203,11cyclictest35966irq/58-eth007:30:143
1603999201,15cyclictest35966irq/58-eth011:35:133
1603999201,15cyclictest35966irq/58-eth011:35:133
16039992011,5cyclictest0-21swapper/312:15:203
1602599208,9cyclictest14716-21cron08:50:022
1602599202,9cyclictest3751-21latency_hist10:05:012
1602599201,16cyclictest12750-21timerwakeupswit08:10:282
1602599201,11cyclictest22579-21sed07:50:192
1602599201,10cyclictest736-21tail09:25:282
1602599201,10cyclictest17170-21kthreadcore07:10:202
1601299202,13cyclictest20446-21grep09:20:181
1601299201,3cyclictest0-21swapper/108:20:211
16012992012,5cyclictest6282-21ssh09:30:181
1601299201,13cyclictest11489-21ssh10:45:141
1601299200,17cyclictest4114-21find10:05:111
1599799206,12cyclictest0-21swapper/008:20:120
1599799204,3cyclictest131rcu_preempt07:40:260
1599799203,4cyclictest12-21ksoftirqd/008:30:170
1599799203,4cyclictest12-21ksoftirqd/008:30:160
15997992013,4cyclictest5127-21sh08:40:000
1599799201,14cyclictest0-21swapper/007:45:100
1603999199,7cyclictest35966irq/58-eth011:20:133
1603999194,9cyclictest0-21swapper/311:50:123
1603999192,7cyclictest0-21swapper/308:15:123
1603999192,12cyclictest0-21swapper/311:40:143
16039991912,5cyclictest35966irq/58-eth012:35:123
16039991910,6cyclictest0-21swapper/310:05:253
1602599191,10cyclictest4283-21sort08:05:142
1601299199,8cyclictest23345-21apt-get08:25:121
1601299198,4cyclictest131rcu_preempt07:20:001
1601299192,13cyclictest1491-21cron09:10:021
16012991911,5cyclictest18788-21grep08:50:281
1599799194,3cyclictest0-21swapper/008:15:140
1599799193,6cyclictest12-21ksoftirqd/007:25:230
1599799193,3cyclictest12-21ksoftirqd/007:10:140
1599799193,10cyclictest0-21swapper/008:05:120
1599799192,5cyclictest12-21ksoftirqd/007:50:280
1599799191,5cyclictest29484-21fschecks_count09:05:130
1599799191,15cyclictest7784-21ls07:35:140
1599799190,5cyclictest12-21ksoftirqd/008:25:200
1603999188,3cyclictest0-21swapper/311:03:033
1603999183,12cyclictest35966irq/58-eth010:25:123
1603999182,12cyclictest0-21swapper/310:35:123
1603999182,10cyclictest0-21swapper/309:05:113
1603999182,10cyclictest0-21swapper/307:55:143
1603999182,10cyclictest0-21swapper/307:45:113
1603999182,10cyclictest0-21swapper/307:10:153
16039991811,4cyclictest0-21swapper/310:15:113
16039991811,4cyclictest0-21swapper/309:27:283
1603999181,13cyclictest0-21swapper/308:35:203
1603999181,12cyclictest0-21swapper/307:15:273
16039991810,6cyclictest0-21swapper/312:20:093
16039991810,3cyclictest0-21swapper/312:25:273
1601299182,11cyclictest5153-21awk08:40:001
16012991812,4cyclictest0-21swapper/112:00:501
16012991811,4cyclictest0-21swapper/109:15:031
1601299181,10cyclictest20406-21kthreadcore08:55:191
1603999179,6cyclictest0-21swapper/312:05:073
1603999179,6cyclictest0-21swapper/311:55:063
1603999179,6cyclictest0-21swapper/309:31:203
1603999173,10cyclictest0-21swapper/312:00:123
1603999172,10cyclictest0-21swapper/308:25:133
1603999172,10cyclictest0-21swapper/307:35:143
1603999172,10cyclictest0-21swapper/307:25:133
1603999171,13cyclictest0-21swapper/310:10:153
1603999171,12cyclictest0-21swapper/312:10:183
1603999171,11cyclictest0-21swapper/307:50:223
1602599171,13cyclictest16090-21latency_hist07:10:022
1601299178,6cyclictest0-21swapper/107:10:251
1601299177,7cyclictest7953-21wc07:35:181
1601299175,6cyclictest0-21swapper/110:15:131
1601299172,12cyclictest32636-21kthreadcore07:25:201
1601299172,11cyclictest27429-21irqcore07:55:171
1601299171,8cyclictest28191-21cpu08:30:141
1601299171,8cyclictest28191-21cpu08:30:131
1601299171,14cyclictest7558-21latency_hist07:35:011
1601299171,14cyclictest12878-21kthreadcore08:45:211
1601299171,12cyclictest26999-21tr09:00:211
1601299170,5cyclictest23169-21cat07:50:201
1599799173,4cyclictest12-21ksoftirqd/007:30:290
1603999162,9cyclictest0-21swapper/308:45:133
1603999161,9cyclictest0-21swapper/308:50:153
1601299169,5cyclictest26074-21irqcore07:20:171
1601299161,12cyclictest13767-21apt-get08:15:121
1601299161,11cyclictest4244-21diskstats08:05:141
1599799163,9cyclictest16048-21cat07:10:010
1603999151,9cyclictest0-21swapper/308:05:273
1603999151,11cyclictest0-21swapper/309:00:283
1601299143,9cyclictest16078-21latency_hist07:10:011
1603999133,8cyclictest0-21swapper/308:25:013
1603999131,8cyclictest0-21swapper/307:20:133
1603999125,3cyclictest0-21swapper/308:10:123
1603999101,7cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional